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ABSTRACT 

Currently, the two most commonly used night optical devices employed in 
military operations are the long-wave infra-red sensor and the image intensified sensor. 
Recent advances in technology have permitted the fusion of the output of these two 
devices into a single color display that potentially combines the capabilities of both 
sensors while overcoming their limitations. Although the concept is appealing, previous 
sensor fusion studies have been inconclusive on the benefits of an artificially colored 
target. Perhaps, an artificially colored target disrupts an operator’s visual processing 
thereby hindering the detection of a target. The purpose of this thesis is to compare the 
effects of artificial color, natural color, and monochrome formats in visual scene 
perception. It is hypothesized that participant response times and error rates would be 
greater at detecting an artificially colored target compared to a natural colored or a target 
presented achromatically. Two experiments were conducted. Experiment 1 used non- 
degraded imagery and Experiment 2 used degraded imagery to compare these effects. It 
was found that reaction time and error rates for naturally colored and achromatic images 
were similar and substantially less when compared to artificially colored images. For 
degraded scenes, natural color was more beneficial when compared to achromatic and 
artificially colored scenes. Additionally, artificially colored scenes caused extremely 
large error rates and reaction times. These results will provide algorithm developers 


insight into the importance of color constancy. 
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EXECUTIVE SUMMARY 

Recently, military requirements have determined the need to improve existing 
night vision devices used in military applications. Shortfalls currently exhibited by these 
Operational night vision devices are the lack of target and environment contrast, poor 
display resolution, inadequate magnification properties and small fields of view. 
Technology now exists that can provide a modification to existing night vision systems 
by replacing them with a dual-band color fused sensor. Modifying these systems is more 
cost effective than the complete replacement. Incorporating the features offered by the 
dual-band color fused sensor into current military hardware may significantly improve 
the aforementioned limitations. Extensive research in visual search experiments has 
demonstrated the utility of sensor fusion; however, the advantages are not clearly defined. 
It has been determined that color fusion may benefit target detection but hinders a 
participant’s situational awareness. Evaluation provided by various operators have 
indicated that the lack of color constancy of the fused devices caused the scenes to be 
aesthetically displeasing. 

Vision scientists have long hypothesized that color enhances visual performance 
in object recognition. Studies have shown that color plays a role in object recognition, 
but the significance of color has often been debated. Color is commonly used to segment 
images into regions that contain information about specific objects within the scene. 
However, it has generally been demonstrated that object recognition is based primarily on 
the representation of shape and that identification by surface texture and color may 
therefore be used as secondary cues in identification. With this theory in mind, we must 


ask why the military would want to incorporate color into existing night vision devices. 


XV 


Space requirements in military vehicles and aircraft have limited the size of the 
visual displays. These small displays present imagery that is often blurred or viewed 
under low light conditions. Images that suffer from this form of degradation lose their 
visual acuity and edge definition. Research concerning colors role using degraded 
objects has demonstrated that color is important in object identification. Without defined 
object contour, color is used to segment objects from their background to facilitate 
identification. 

One significant limitation attributed to night vision fused sensors is that current 
algorithms are incapable of providing consistent color to objects. This is because 
information taken from night sensors is normally outside the range of the HVS's 
sensitivity. Therefore, color is added according to the existing contrast between the 
object and background temperature differential. The temperature differential dictates the 
coloring scheme of the object, thereby prohibiting any form of standardization amongst 
identifying objects by specific colors. However, false coloring algorithms are currently 
widely used in medical and meteorological displays. These diverse areas have devised 
color schemes as symbolic measures of significance. With the inability to standardize 
object color in fused night vision sensors, will color enhance object recognition? 

The influence of color was studied as a feature against its influence as stored 
knowledge in object recognition. Stored knowledge is interpreted as semantic 
information that explains the prototypical colors of objects, such as the knowledge that 
apples are typically red or green. Participants respond faster to objects having 
prototypical colors than objects that do not possess any correlation to the presented color. 


When objects were viewed in an unnatural scene, participant reaction times increased. 


XV1 


This thesis is designed to validate previous color studies, by using professional 
photographs with and without degraded edges. It is hypothesized that in high resolution 
scenes, natural color images will not. enhance object recognition when compared to 
monochromatic images, and non-prototypical (artificial) colored scenes will hinder object 
recognition. For low-resolution images, natural color will enhance object recognition 
when compared to monochrome scenes, and artificially color scenes would hinder object 
recognition. 

The results of both experiments confirm the hypothesis. Participant’s that viewed 
high resolution natural colored and monochromatic scenes had similar reaction times and 
error rates. In the artificially colored scenes, the reaction times and error rates were 
substantially larger. For low-resolution scenes, natural color significantly enhanced 
object recognition while artificial color hindered a participant's ability to efficiently 
recognize the objects. Again, participant error rates paralleled their reaction times. 
Participants were more efficient when viewing natural color scenes and were 
significantly inefficient when viewing scenes that did not possess any form of consistent 
color. 

This research has confirmed previous vision studies that included the use of 
objects being observed in their natural environments instead of against a neutral 
background. It is recommended that similar vision studies be conducted using imagery 
taken from color fused sensors instead of commercial imagery to further validate these 


findings. 
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I INTRODUCTION 


Throughout the history of warfare, nations have pursued methods that helped 
them gain tactical advantage over the enemy. Many of these methods involved the 
element of surprise. However, because of advancements in satellite surveillance systems 
and other forms of innovations in intelligence gathering, the element of surprise has been 
effectively reduced. A remaining element of warfare that has yet to be fully exploited is 
the ability to operate under the cover of darkness, or in low visibility. Most intelligence 
and satellite surveillance systems parallel the human visual system (HVS) in that, they 
educe reflected light, and as such, are ineffective at night. Therefore, military technology 
has enunciated the development and refinement of night vision systems that permit forces 
to operate under the cover of darkness. These systems are commonly referred as Night 
Vision Devices (NVD's). 

The two most common NVD's employed by military forces are the forward- 
looking infra-red radar (FLIR) and the night vision goggle (NVG). Although their 
purpose 1s to expose environmental information in the darkness, they function very 
differently. FLIR's operate in the infra-red region of the electromagnetic spectrum. They 
detect thermal differences between an object and its background. The greater the thermal 
difference, the higher the contrast portrayed in the image. Hence, a major limitation of a 
FLIR sensor is that it requires a thermal difference to adequately reveal a target that is 
detectable to a human observer. There are numerous occasions where this limitation may 
impede night operations. Heavy precipitation and dense foliage tend to create a 
homogeneous thermal scene. When this occurs, FLIR sensors are superfluous to night 


operators (MAWTS-1, 1995). 


NVG's require reflected radiated light to function. Whether this energy originates 
from celestial radiated energy or man-made energy, NVG's magnify this energy to 
illuminate the scene. A major limitation of this device is the requirement of reflected 
energy. Operations under cloud cover or in dense foliage may severely restrict the 
devices potential (Wolfe, 1994). 

U.S. military forces have successfully employed these devices in combat 
operations. As recently as in Operation Desert Storm and Bosnia peacekeeping 
operations, military forces have used NVD's frequently and have conducted extensive 
night operations safely and effectively (Time-Life Books, 1991). Unfortunately, the 
widespread proliferation of this technology (Klass, 1994) has permitted other nations to 
acquire these devices which has reduced the technological superiority of U.S. forces on 
the battlefield. In order to regain battlefield superiority, the United States is pursuing 
technology that may enable forces to reacquire this advantage. Some of these advantages 
may come from improving the existing night vision capability. 

Computer technology has permitted the U.S. to consolidate sensory outputs from 
a cones of different NVD's within a single representation. One such combination 
consists of fusing the output of the NVG sensor with the output of the FLIR sensor. 
Individually, each sensor displays unique characteristics of the visual scene. Despite the 
deficiencies of each sensor, fusing individual sensor outputs into a single representation 
may potentially overcome individual weaknesses and provide more opportunities to 
observe the scene under more heterogeneous environmental conditions. For example, in 
the absence of visible light, an object may still be observed if it possess any form of 


thermal signature. Although sensor fusion may potentially take advantage of individual 


sensor limitations, the output of each sensor is presented in the same monochromatic 
format that is displayed with individual sensors. Because of this, a lack of background 
contrast may continue to exist. One possible method of subjugating this lack of 
background contrast is by introducing color to the display. Color may intensify the 
context of a scene, thereby permitting the operator more opportunity to efficiently 
perform his search (Krebs, Scribner, Miller, Ogawa, and Shuler, 1998). 

These color fusion algorithms presume that the addition of color to the scene 
would enhance object detection (Scribner, Warren, Schuler, Satyshur and Kruer, 1998). 
One significant limitation to providing color to fused images is the inability to reconstruct 
color constancy. Current computer algorithms provide image colors according to thermal 
contrast. Because the thermal signature is continuously changing, the inability to control 
for color constancy is lost. Recent studies comparing the possible benefits of employing 
color in displays with monochromatic displays have been inconsistent. These studies 
either failed to show a significant advantage for color over grayscale images or they 
found the advantage to be scene dependent. Therefore, the research failed to find specific 
advantages to displaying fused color scenes. (Sampson, Krebs, Scribner, and Essock, 
1996; Krebs, et al., 1998; Essock, Sinai, McCarley, Krebs, and DeFord, 1998) 

Similar false coloring algorithms are currently being used in medical (Schmidt, 
Hier, Benyamin, and DeForest, 1988), meteorological (Uttal, Baruch and Allen, 1994), 
and automotive displays (Krebs, Scribner, Schuler, Miller and Lobek, 1996). These 
diverse areas have devised color scenes as symbolic measures of significance. For 
example, the color red displayed on a weather radar screen portrays the existence of 


severe precipitation whereas the green may signify light precipitation. Research and 


training in these fields have standardized these false colors into providing meaningful 
information to individuals. It is possible that if a similar standardization of color is 
incorporated in image fusion algorithms, the meaning of specific colors could be learned 
and therefore proven beneficial. 

NVD’s are single-band sensors that operate outside the range of the human visual 
system's sensitivity. They convey no chromatic information. Current artificial color 
algorithms base their coloring schemes on sensor contrast between the reflective and 
thermal energy detected from the scene (Krebs, et al., 1998). The objective of artificial 
color processing 1s to emphasize sensor contrast rather than reproducing the actual or true 
colors of the scene. As a result, the “correct” color information of the scene is lost. 
Furthermore, the sensor contrast differential dictates the coloring scheme of the image, 
thereby prohibiting the ability to provide standard colors to objects within a scene. 
Perhaps it is this lack of color standardization that is a contributing factor to the 
inconclusive results of previous psychophysical fused color studies (Steele and Perconti, 
1997). 

Basic research on object recognition has determined color to be non-beneficial 
when compared to grayscale images (Biederman and Ju, 1988). However, this inference 
has often been debated. Some research has shown that color benefits object recognition 
while others have not shown any significance. A_ possible reason for these 
inconsistencies may arise from the differing experimental methodologies employed by 
these studies. It may be the type of task performed and the cues that were provided to 
participants that determined the role of color (Wurm, L.H., Legge, G.E., Isenberg, L.M., 


and Luebker, A., 1993; Biederman and Ju, 1988; Joseph and Proffitt, 1996). No known 


research has attempted to determine colors role using global scenes in object recognition. 
Therefore, attempting to apply the existing color theories to pseudo-color fused theory 
may be inappropriate. 
A. OBJECTIVE 

The objective of this thesis is to clarify the role of natural color and artificial color 
in natural scene perception. Currently, visual devices in military vehicles are normally 
small, compact and are often characterized by poor resolution. This may cause imagery 
to be presented as blurred and lack defined contour information. Under these additions, it 
might be expected that color would benefit object recognition. This leads us to question 
the function of color vision, and the role it plays in object recognition, especially if the 
displayed color does not depict the natural scene. 


B. THE RESEARCH QUESTION 


A major reason that previous research concerning the role of color in object 
recognition has been inconclusive may be due to the existing experimental 
methodologies. The methodologies of previous research ranged from comparing black 
and white line drawings to colored pictures of manufactured objects (Biederman and Ju, 
1988), to comparing natural and artificially colored line drawings of natural objects 
against neutral backgrounds (Joseph and Proffitt, 1996). It may be concluded that 
different methodologies may produce varied results. In this thesis, objects will be 
observed in their natural settings with the intent to compare the Sis of natural and 
artificial color images using professionally photographed images. Once participant 
performance is known when viewing non-degraded color images, it is expected that the 


results will adversely differ when viewing degraded images. 


ce SCOPE, LIMITATIONS, AND ASSUMPTIONS 


For this research, professionally photographed daylight images were used in place 
of fused images to determine the role of true and artificial color in object recognition. 
This provided more flexibility in selecting images based upon standards that were set 
prior to data collection. Although the selected images did not resemble fused imagery 
from NVD’s, the role of true color and artificially colored natural scenes could still be 
examined. Two experiments were conducted. The first experiment attempted to 
determine whether artificial color was perceived similarly as monochrome and natural 
color using non-degraded imagery. Based on previous findings, it is hypothesized that 
there will be no advantage of color in object recognition, but artificially colored scenes 
may disrupt the HVS's ability to efficiently perceive objects within the scene. The 
second experiment attempted to simulate NVD representations using similar commercial 
images used in the first experiment. However, noise was added to the imagery to 
approximate the degraded resolution of NVD displays. To validate the research 
conducted by Biederman and Ju (1988), Joseph and Proffitt (1996) and Wurm, et al., 
(1993), it is hypothesized that there will be an advantage of color in object recognition 
and that artificially colored images will disrupt the HVS when observed in degraded 
conditions. Additionally, given the additional data obtained in the second experiment 
using different imagery, portions of the findings from the first experiment could be 
validated. 


D. ORGANIZATION OF STUDY 


This thesis is organized into five chapters. Chapter I is the introduction, which 
gives a general description of current limitations to providing color to fused images. 


Chapter II provides the necessary background information relevant to night vision 


technology. It also provides in detail, previous research of the role of color in object 
recognition. Chapter III and IV describe the methodology used to conduct Experiments 1 
and 2 respectively, the procedures used_in selecting and manipulating the images, and the ~ 
experimental design. The chapters also provide data analysis and interpretation of the 
data collected from both experiments. Chapter V_ provides conclusions and 


recommendations for further research. 
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II. BACKGROUND 


A general historical review of events leading up to the technology that produced 
these devices should provide the reader with a basic understanding of the problems 
associated with providing color to NVD’s. A general description and characteristics of 
individual sensors is provided. With knowledge of sensor capabilities and limitations, the 
characteristics of sensor fusion can be understood. Also, a comprehensive understanding 
of previous research involving the use of color in object recognition is provided to aid in 
the development of theories that may help provide answers to the contradictions exhibited 
in previous color fusion studies. 


A. HISTORY 


To be successful in warfare, a striking force must maintain an advantage over the 
adversary (Clausewicz, 1976). Efforts to maintain this advantage have resulted in the 
development of more destructive weaponry and the creation of more efficient tactics. 
Although many new technologies were introduced during World War I, American forces 
did not exploit night combat operations until the latter stages of the war. Until then, night 
operations were typically utilized for planning subsequent daytime operations and 
relocating forces for an ensuing battle. Many U.S. forces were lost in combat during 
daylight hours. It was not until the “Battle at Cape St. George” during the Pacific 
campaign in World War II, where Captain (later Admiral) Arleigh Burke’s forces 
inflicted heavy damage upon the main Japanese surface forces, that large-scale night 
tactics were designed and practiced. One of the major contributing factors to Burke’s 
success was his ability to exploit night operations. He recognized this paradigm and 


prepared his forces by devising and employing tactics that took advantage of the enemy’s 


inability to respond under these conditions (Potter, 1990). 

The Korean War produced no radical changes to combat tactics during night 
operations. This was partially due to the insufficiency of technical advances of night 
optical devices, and the fairly limited understanding that researchers had of human visual 
processing. However, since then, the infra-red industry has expanded rapidly. This 
industrial development was motivated by the inability of U.S. forces to expostulate North 
Vietnamese forces from conducting night operations against isolated outposts, force 
resupply, or moving logistics (Schwarzkopf, 1992). Since the post Vietnam era, virtually 
all military high value platforms possess infra-red imaging capability. The vast 
investment in NVD’s by the military services and law enforcement agencies has spawned 
an enormous industrial community dedicated not only to the manufacture of the devices, 
but also to their large-scale integration into weapons, ships and combat vehicles. As a 
result, researchers who analyze, measure and predict the performance of NVD’s have 
grown from a small community, intimately involved and thoroughly familiar with the 
technology and its theory, to an extremely diverse population of engineers, managers and 
scientists. The necessity to see “into the night” is the driving force in the development of 
NVD’s (Berry, 1993). 

There are numerous night vision enhancement systems in the U.S. armed forces’ 
inventory. Every system is capable of functioning beyond the HVS's unaided capability. 
These systems have been proven effective in a myriad of diverse combat operations. 
However, unanticipated problems have arisen while utilizing these devises. A human’s 
unaided perception of the surroundings at night is vastly different when observed with 


NVD’s. The user’s lack of understanding of the night environment and its impact on the 
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NVD’s performance has caused the capabilities of these devices to be exceeded, resulting 
in numerous mishaps (Salvendy, 1997). 

Currently, almost all military personnel have trained with or have been exposed to 
NVD’s. With the radical change in military budgets and huge reduction of forces, the 
exploitation of night operations allows U.S. military forces to do “more with less” while 
retaining the combat advantage over enemy forces. As recently seen in Operation Desert 
Storm, many of U.S. military campaigns occurred at night. Coalition forces exposed 
Iraqi forces to a “24 hour battlefield” by employing NVD tactics; a strategy the Iraqi 
forces were ill prepared to confront. U.S. forces easily demonstrated the ability to 
conduct sophisticated and coordinated operations with the aid of these devices on their 
platforms (Schwarzkopf, 1992). A brief overview of the NVD’s is presented to provide 
the reader with a basic understanding of these devices. A more detailed coverage of the 
theory and operation of NVD’s is provided in MAWTS-1 (1994). 


B. IMAGE INTENSIFIERS (NVG’S) 


NVG’s operate in the near visible and near infra-red (IR) spectrum (wavelengths 


of 570-900 nanometers) and depend entirely on reflected energy (Figure 1). 
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Figure 1: The portions of the electromagnetic spectrum used in unaided human 
vision, NVG's and FLIR's. (MAWTS-I, 1995) 


The first generation NVG's relied entirely on light at visible wavelengths, but the 
third generation (GEN-II) instruments make use of infra-red light just beyond the visible 
spectrum (NVESD, 1997). Current NVG technology centers on the GEN-II devices. 
Even on the darkest of nights when visible light is practically non-existent, infra-red 
energy is still plentiful. Additionally, the reflective characteristics of different surfaces 
are more pronounced in the infra-red band, thus creating an image displayed with greater 
contrast (Time-Life Books, 1991). 

The two sources of light that enhance the HVS aided by NVG’s are reflected and 
generated light. Reflected light normally originates from energy emitted by celestial 


bodies such as the moon or stars that is reflected off an object's surface. Generated light 


originates from artificial sources of illumination such as city lights and shipboard lights. 
A specific characteristic of NVG's is that with proper ambient light levels, the visual 
clarity is superb. There are several_variables that affect the optimal performance of 
NVG’s. These variables include the amount of illumination within the scene, 
characteristics of the terrain, and atmospheric conditions in which the sensor is operating 
(FM 34-81-1, 1992). Atmospheric conditions have the greatest impact on NVG 
effectiveness. The dominant atmospheric factor that affects NVG performance is the 
quantity of water vapor (humidity) present. This is because water molecules in the air 
absorb and reflect light from the intended source (FM 34-81-1, 1992). A sample image 


viewed with a NVG 1s shown in Figure 2. 





Figure 2: Sample image taken with a NVG. Notice the clarity of the objects. 
(A.P.Hill, 1998) 


C. THERMAL SENSORS (FLIR) 


FLIR’s are electronic devices that convert invisible thermal energy into a visible 


image. FLIR’s operate in the 7-14 micron wavelength band of the electromagnetic 
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spectrum (Figure 1). FLIR technology is based on the detection of emitted and reflected 
thermal energy. FLIR’s operate as differential temperature measuring systems, in that, a 
measurable temperature difference must exist between an object and the background in 
which the sensor operates. This measurement is referred to as “DELTA T” 
(Schlessinger, 1995). 

The infra-red signatures of targets and backgrounds are far more complex than the 
visible signatures in NVG’s because emission has virtually no influence (NVESD, 1997). 
As complex as infrared signatures are, the limited scientific capabilities at the disposal of 
the FLIR community cannot cope with anything beyond the simplest expression of target 
and background signature. The predominant analysis in thermal theory is currently based 
on the assumption that an object behaves like a blackbody at a temperature some number 
of degrees different from a uniform blackbody background, such as the environment in 
which the object is viewed (Campana, 1993). 

The two general types of sources that enable FLIR devices to function are thermal 
and selective radiated thermal energy. Thermal sources, such as the sun or combustible 
engine, radiate over a wide frequency spectrum with the maximum radiated energy level 
concentrated at a particular frequency. Selective thermal sources generally concentrate 
on a very narrow frequency band. An example of selective thermal radiation is a laser 
beam (Steele and Perconti, 1997). 

The diurnal cycle is a measure of the rate at which objects and their backgrounds 
heat and cool. Large dense objects tend to heat and cool slowly, whereas, less dense 
objects such as trees and grass, heat and cool more rapidly. Objects heated by sunlight 


are the mechanism through which the diurnal cycle drives thermal signatures (Shumaker, 
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Wood, and Thacker, 1988). The rate of change in temperature of a surface is dependent 
not only on its emissivity but also on its shape, size, heating capacity, conductivity and 
convective cooling/heating effects (Wolfe, 1996). 

Sometimes, the radiated temperature of an object matches the temperature of the 
surrounding environment resulting in a Delta T = 0. On hot days, an object normally 
fails to reach the evening background temperature before the sun rises again to restore the 
heat cycle. As the sun mises, the cooler background warms up rapidly and soon exceeds 
the target temperature. Twice in a 24-hour period, Delta T goes below zero. From 
sunrise to late afternoon, the background temperature is generally greater than the object 
temperature thereby masking the objects thermal effect on a FLIR device (FM 34-81-1, 


1992). (See Figure 3) 
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Figure 3: The Diurnal cycle for a man-made object and background terrain. 
Crossover times are also shown (MAWTS-1, 1994) 


Atmospheric conditions also have a very profound effect on the effectiveness of 
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FLIR’s. The two predominate atmospheric factors are energy absorption and scattering. 
Thermal absorption in the atmosphere is a greater concern that contributes to thermal 
Signature loss and scatter (MAWTS-1, 1994). A characteristic of FLIR representation is 
the amount of contrast level between objects and backgrounds. This contrast dictates the 


resolution of the display. A sample image viewed with a FLIR 1s shown in Figure 4. 
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Figure 4: Sample image taken with a FLIR. Notice the brightness and poorer 
resolution of the objects. (A.P.Hill, 1998) 


Current FLIR technology is centered on first generation (GEN-I) FLIR devices. 
The U.S. Army began integration of second generation FLIR’s into new and existing 
weapon systems to maximize U.S. forces advantage on the battlefield. The center piece 
of this system is a common subsystem that is currently being produced for the Abrams 
Tank, Bradley Fighting Vehicle, and Long Range Advanced Scout Sensor System 
(NVESD, 1997). 


D. FUSED SENSORS 


Improvements to individual NVD's have provided substantial increases in sensor 


detection ranges and display resolution. However, these improvements cannot avoid the 


16 


limitations of the electromagnetic spectrum. Each sensor suffers from its own 
limitations. NVG’s must have reflected light for detection and FLIR's must be able to 
detect a contrast in thermal energy between an object and the background. Scene 
analysis by an operator may potentially benefit from a fused representation (combined 
FLIR and NVG) of a scene taken in different spectral bands. For example, following a 
period of extensive cooling such as a rain shower, the visible bands may represent the 
background in great detail whereas the infra-red bands that represent the scene may be 
negligible or provide a representation of the scene that is severely degraded due to the 
low thermal contrast. Also, a target that is camouflaged for visible detection may not be 
detected in the visible bands, but may be clearly represented in the infra-red bands by 
providing a thermal contrast between the object and background. The fusion of visible 
and thermal imagery on a single display may allow both target detection, provided by the 
thermal image with respect to the context, and unambiguous target localization, provided 
by the visible image (Toet, IJspeert, Waxman, and Aguilar, 1997). Technology now 
exists that enables information provided by these sensors to be combined and represented 
on a single display. 

The basis for fusing multiple sensors originated from the notion that pit vipers 
incorporate a similar concept. Various species of pit vipers combine the attributes of 
visible and infra-red vision for hunting at night with little or no visible light. The snake’s 
visual system is composed of infra-red sensors and pit organs, locate near the head, that 
open below and in front of the eyes. Infra-red information is sensed by the snake’s pit 
organs and is then sent to the brain. where it is combined with visible information 


obtained by the snakes eyes (Hartline and Newman, 1982). This combined information 
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allows the pit viper to hunt efficiently in daytime as well as at night. Although the pit 
viper can strike with deadly accuracy on just the thermal signature of the prey, mapping 
of neurological impulses in the brain show that the visual and infra-red spectrum are 
combined (fused) to provide a complete picture of the scene (Hartline and Newman, 
1982). 

Having the capability to combine multiple NVD’s contained in military 
equipment, ongoing research is being conducted to attempt to develop an optimum multi- 
sensor integration suite (Campana, 1993). Some advantages of combining multiple 
spectrums into a single representation might be: 

e Reduced cost, space, and weight requirements from combined resources. 


e Reduced operator workload by eliminating the need to alternate between two 
Sensors. 


e Improved object search, detection, and recognition. 

One of the characteristics of the HVS is its ability to process color efficiently 
(Jayle, Ourgurd, Baisinger, Holmes and Duke-Elder, 1959). Presenting displays in 
monochromatic format, such as the displays of NVD’s, restricts the capability to 
optimally utilize the HVS. If the HVS were monochromatic, scenes would be similar to 
those displayed on a cathode-ray tube (CRT), where digitized monochromatic images 
physically represent each pixel as one of 256 different levels of brightness (gray-levels) 
(Waxman, Fay, Gove, Seibert, Racamato, Carrick, and Savoye, 1995). However, the 
HVS can perceptually distinguish only about 100 different gray levels on a CRT with 
images collected under ideal lighting conditions (Scribner, et al., 1998). For images 
collected under dim light, the gray-scale depth of the image is even more limited (Toet 


and Walraven, 1996). To improve the probability of detecting an object within a scene, 
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combining multiple bands into a single fused color scene should enhance the contrast 
between the foreground and background objects in the scene (Scribner, Satyshur and 


Kruer, 1993). Figure 5 shows an example of an image viewed under fused color. 





Fipare 3: Samco image a with rruseds sensors A. P. Hill, “1558. 


Besides the advantages already discussed of displaying fused sensors on a single 
representation, fusion also provides the opportunity for the presentation of imagery on 
chromatic displays. However, color imaging is only possible when the responses of the 
fused sensors differ in spectral sensitivity (Krebs, et al., 1998). 

It is important to understand the distinction between a color scene observed by the 
unaided eye through the human observer and a processed pseudo-color scene. 
Psychologically, certain colors have developed meaning with time. Red for instance, 
possesses an instinctive psychological impact that tends to draw attention, enhances 
detection and discriminates possible dangers (Valdez and Mehrabian, 1994). In order to 
ensure that these color cues appear the same for a wide range of environments throughout 
the day, the visual system adapted the ability to distinguish these colors under varying 


natural lighting conditions (Sekuler and Blake, 1994). 
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The spectral composition of light impinging on the HVS sensory receptors is 
determined by the spectral composition of the luminance and the intensity of the energy 
of an object's surface reflection. The intensity of the reflected light is extremely 
inconsistent, varying due to factors such as time-of-day effects, atmospheric conditions, 
and the amount of airborne particles in the air. This lack of consistency for the intensity 
of reflected light required the HVS to develop a variety of mechanisms to disentangle the 
contradictions of varying lumination and thereby to achieve nearly constant color 
perception based on distal surface reflectivity (Matlin and Foley, 1997). A major 
limitation to sensor fusion systems is that these mechanisms cannot be duplicated to 
achieve the same constant color perception. Sensor fusion systems produce color 
mappings that vary with changes in the intensity of light's spectral composition. The 
potential disadvantage caused by the variability in spectral illumination may be 
magnified by the relatively arbitrary color mappings produced by this new technology. 
This result is due to the spectral sensitivity channels that compose sensor fusion 
technology that do not correspond to the trichromatic channels of the HVS. This causes 
the color mapping produced by fused sensors to be unnatural in appearance (Krebs, 
1998). Before proceeding with a discussion on previous natural and false color studies, a 
more detailed description of color vision may aid in understanding the difficulties in 
relating false color to natural color. 


EK. HUMAN VISUAL SYSTEM 


The HVS is an extremely complicated system. It is made up of two separate but 
inter-related systems; the eye (receptor) and the optic neural pathways (processors) 
(Legrand, 1968). The sensory receptors in the eye are sensitive to energy within a limited 


range of the electromagnetic spectrum, specifically, wavelengths from approximately 0.4 
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to 0.7 microns. Figure 6 depicts the electromagnetic spectrum. Any energy beyond this 
range will not be visually detected because it has no effect on the receptors (Sekular and 


Blake, 1994). 
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Figure 6: The electromagnetic spectrum. (Matlin and Foley, 1997) 

Visual processing is accomplished by a system of neurological pathways between 
the retina and the brain that convert the reflected light energy into images. A more 
detailed description can be found in textbooks on vision (Sekular and Blake, 1994). 

The retina is the interface between perception and reception in the eye. It 
contains two major types of photoreceptor cells called rods and cones. Rods are 
primarily involved in vision under very low levels of light. They are slow acting, provide 
minimal visual clarity, and cannot discern color. Rod vision is also referred to as scoptic 
vision (Sheppard, 1968). There are three sub-types of cone cells that are fast acting, 


sense the entire visual color spectrum, and provide maximum clarity, but function poorly 
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during low light conditions. Cone vision is also referred to as photopic vision (Legrand, 
1968). 

Each photoreceptor cell is coated with a chemical substance called photopigments 
that accomplish the transduction of light. For the three sub-types of cone cells, each 
photopigment absorbs more light in one portion of the electromagnetic spectrum than in 
any other area (Matlin and Foley, 1997). Estimates of the portions of the electromagnetic 


Spectrum that are absorbed by the cone cells are shown in Figure 7. 
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Figure 7: Spectral Frequencies of the L, M, S cones. (Wandell, 1995) 
As seen in Figure 7, each particular cone cell is referred to as the S-cones 
(shortwave), M-cones (mediumwave) and L-cones (longwave) and each corresponds to a 


particular color with the color spectrum, blue, green and red, respectively (Wandall, 


Jip 


ISD): 

The spectral properties of light are measured in terms of the amount of energy 
emitted at each wavelength, namely the spectral power distribution of the light source. 
The portion of light that is reflected off an object's surface defines the surface-reflectance 
function. This reflected energy leads to different amounts of absorption in the three cone 
classes and it 1s this interpretation of cone absorption that is the basis of color perception. 
Therefore, the light incident to the HVS is a function of the spectral power distribution of 
the ambient illumination and the surface-reflectance function (Wandall, 1995). For 
example, suppose that the eye was presented a target color (orange) of 600-nanometer 
wavelength. From Figure 7, the target would present a large amount of absorption in 
both the M-cones and L-cones and the color would consist of a mixture of L-cones (red) 
and a lesser amount of M-cones (green) that produces the color orange. The remaining 
portions of the reflected energy are absorbed by the object, prohibiting the receptors from 
receiving energy within the specific frequency bands (Matlin and Foley, 1997). 

Isaac Newton demonstrated in the late 17" century that white light consists of a 
combination of different colored lights that are evenly distributed across the visible 
electromagnetic frequency spectrum (Matlin and Foley, 1997). Figure 8 shows the 


primary colors and associated wavelengths in relation to the electromagnetic spectrum. 
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Figure 8: Visible Color Spectrum (Matlin and Foley, 1997) 


It is difficult to imagine that objects may look as though they are colored when in 
fact, they are actually reflecting light from the unabsorbed portions of the spectrum. A 
chair may appear blue, but in reality it is blue because its surface is absorbing most of the 
medium and long wavelengths and is primarily reflecting the wavelengths from the blue 
portion of the spectrum to the HVS. Since the S-cones would absorb the majority of the 
energy and the L-cones and M-cones absorption rates may be negligible, the object would 
be perceived as blue in color (Matlin and Foley, 1997). 

Another characteristic of color is color constancy. In color constancy, the HVS 
tends to see the same colors during varying levels of illumination. This is because as the 
illumination changes, the sensitivity distribution of a light source will change at a similar 
rate across the visible frequency spectrum. The distribution of wavelengths reaching the 
photopigments of the cones will receive the same information, except at a different 


intensity. This explains why grass appears to be green on a bright sunny day and on a 
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cloudy rainy day (Sekular and Blake, 1994). 

Light is organized by its wavelength characteristics. As Isaac Newton 
demonstrated, white light that enters a prism is broken into numerous color components. 
A method of organizing all the wavelengths of the visible spectrum is accomplished by 
placing them in a circular position, or color wheel. Figure 9 shows the standard color 


wheel and distribution of frequencies (Matlin and Foley, 1997). 
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Figure 9: Standard color wheel and associated color wavelengths. 
(Matlin and Foley, 1995) 


Color is also characterized by the amount of saturation that is present. A light that 


occupies a very narrow frequency band is said to be pure and highly saturated. An 
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example would be that of a laser beam. Saturation measures the purity of color. As the 
width of the frequency band of the light expands, additional variants of color are added as 
they are received through the respective cone cells and the representation is not as pure. 
Saturation can also be measured on the color wheel. A totally saturated color is displayed 
on the outside edge of the color wheel and unsaturated color is displayed as the 
wavelength approaches the color wheel origin. Saturation is only possible when the 
brightness is intermediate, neither too light nor too dark (Wandall, 1995). Now that an 
overview of the characteristics of color and the HVS has been presented, we can now 
discuss color's role in object recognition. 


F. REVIEW OF THE LITERATURE 


The role of color and its impact on the human perceptual system has intrigued 
vision researchers for many years. Until recently, color has been attributed to object 
recognition, in that it was assumed that color aided an individual’s ability to recognize 
objects (Wurm, et al., 1993). Early studies suggest that color vision evolved in early 
humans to facilitate the search and recognition of food (Polyak, 1957). However, it has 
since been shown that individuals who possess a color deficiency (color blindness) only 
become aware of this deficiency after taking a color test (Steward and Cole, 1989). 
These individuals lead a normal life and function without visual impairment. This leads 
us to question the function of color vision, and the role it plays in object recognition. 
Although no firm conclusion exists concerning the role of color in object recognition, 
most vision researchers agree that color is significant. 

Three tasks have typically been used to determine colors role in object 
recognition. These tasks are classification, verification and naming. Research has shown 


that although color does not influence classification or verification performance, it does 
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influence naming performance (Joseph and Proffitt, 1996). However, it will be later 
shown, that this statement is often debated. For purposes of this thesis, classification is 
defined as objects that belong to a specific category, such as objects that are either living 
or non-living (Davidoff and Ostergaard, 1988). In verification tasks, individuals are 
tasked to validate the relationship between a word and a picture. If an image of an apple 
was presented to a participant, the participant must respond whether the associated word 
or phrase matched the presented object (Biederman and Ju, 1988). 

Biederman and Ju (1988), conducted a significant study on the role of color in 
object recognition. This study compared colored pictures of manmade objects to black 
and white line drawings of the same objects. The objects and line drawings were 
presented in two kinds of tasks. The first task was a naming task. Participants saw a 
Slide of an object that was presented in either a color photograph or line drawing 
depiction. Upon recognizing the object, participants named the object using a voice key 
that recorded reaction times and error rates. It was expected that participant reaction 
times and error rates for the naming task would be smaller for objects depicted in color 
format (Biederman and Ju, 1988). The other task was a verification task. Participants 
had to verify the name of an object by matching it to the image representation. This was 
accomplished by pressing a YES or NO response on a microswitch. Each presentation 
was shown in one of three different exposure times followed by a mask. The exposure 
times were 50, 65 and 100 milliseconds in duration. It was assumed that longer exposure 
duration and slightly dimmer projector intensity would favor colored photography 
(Biederman and Ju, 1988). 


The results of the experiments were surprising. In the naming tasks in which the 
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projector intensity was varied and a mask that immediately followed an image 
presentation was presented randomly, no significant advantages for participant reaction 
times and error rates for the color images were shown. An astonishing finding during the 
naming task portion of the experiment was that reaction times were generally larger for 
images displayed in line drawings when observed with smaller exposure times. This 
suggests that the HVS may respond less efficiently to colored images when the images 
are presented momentarily (Biederman and Ju, 1988). 

The same color photographs and line drawings were used in the verification task 
experiment. Participants were given a verbal one-word description of a target and were 
then instructed to respond once an image was presented. This task enabled the 
participants to form a mental representation of the target and to also anticipate its texture 
and details. The results of the verification tasks mirrored the results of the naming task. 
No advantage for identifying a colored picture over a similar black and white outline was 
found (Biederman and Ju, 1988). 

The researchers concluded that simple line drawings can be identified as quickly 
and accurately as fully detailed, textured, colored pictures of the same object. These 
results support the premise that initial access to a mental representation of an object can 
be modeled by matching the edge-based depiction of the object. Therefore, when an 
object's edge can be perceived by the HVS, color plays only a secondary role in the real- 
time recognition of the object (Biederman and Ju, 1988). 

To parallel Biederman and Ju’s research, another group of visual scientists 
conducted a similar experiment investigating the role of color in categorical judgements. 


Line drawings were also used for the experiments. Coiored and black-and-white 


28 


transparencies were produced for each object that was shown to the participants. The 
objects were rated according to size (small, smaller, large, and larger) compared to a 
trumpet, which was defined as the size standard. Additionally, objects were categorized 
into living and non-living categories. Participants were tasked to verbally indicate the 
specific categories in which the objects belonged. For example, a color presentation of 
an airplane would be placed in the "much larger, non-living" category. Assuming that 
color would have no influence on participant reaction time in the task, the researchers 
were interested in investigating color benefits on image size and recognition categories. 
For example, would a participant recognize a colored depiction of a butterfly or an 
airplane faster when compared to the same pictorial representation in achromatic format 
(Davidoff and Ostergaard, 1988)? 

The results of the experiment showed that color affected neither size nor 
living/non-living classifications. Although color reduced participant reaction times when 
compared to reaction times of monochrome images, the results were not significant. The 
authors speculated that the slightly smaller reaction times for the colored pictures might 
have been attributed to a color brightness advantage (Davidoff and Ostergaard, 1988). 

In a subsequent experiment, identical images were used and participants were 
tasked to name the viewed objects instead of rating them for object size. This procedure 
attempted to determine the relationship of color and naming living and non-living 
objects. Non-living objects typically are not normally associated with a particular color 
and therefore it was hypothesized that the color advantage would enhance living object 
recognition better than non-living object recognition. Previous studies (Davidoff and 


Ostergaard, 1985) indicated that color may be beneficial in object naming but not in 
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object recognition. Their findings however, were based on an inadequate image sample 
size. Therefore, the focus of this naming task was to validate previous findings with 
large image sample sizes. The findings of the research indicated that color did not 
significantly enhance participant reaction time between living and non-living conditions. 
It did have a positive effect on participant's reaction times and supported the previous 
object recognition research but the advantage was found to be not significant. It was 
concluded that color played a role in object naming but has no effect in object recognition 
(Davidoff and Ostergaard, 1988). This conclusion contradicts Biederman and Ju's studies 
that used similar procedures, but different types of pictures. A factor not controlled and a 
possible contention for these inconsistent results was speculated to be the failure of 
Biederman and Ju to control luminance characteristics. It was suggested that visual 
analysis of the color images may have been at a starting disadvantage because of greater 
limitations in edge extraction compared to the line drawings (Wurm, et al., 1993). 

Price and Humphries (1989) examined the effects of color congruency and 
photographic detail on naming and classification of structurally similar and dissimilar 
objects. Using the notion that color does not influence object recognition with edge- 
based effects (Biederman and Ju, 1988), they wanted to determine if these theories could 
be applied to surface-based effects. Surface-based effects predict that it will benefit 
object recognition if the effects are shown in conjunction with their surface details, such 
as a variation in luminance, brightness, texture and color. To determine these effects, the 
authors conducted a series of experiments in which participants performed subordinate 
naming and classification tasks using structurally similar and dissimilar objects. Items 


such as birds were observed with structurally similar shapes and tools for instance, were 
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presented with structurally dissimilar shapes. Price and Humphries re that 
object color and surface detail would be beneficial for discriminating between categorical 
members. They felt that classification of objects could proceed directly from the ~ 
structural description of the object. Their findings supported the hypothesis and revealed 
that surface color benefits not only a naming task but also applies to classification tasks 
when shape information is not sufficient for discriminating between category members. 
They also suggested that surface color is more likely to be important for specific 
categories of objects, namely natural categories, as opposed to artifactual categories. It 
was therefore concluded that the effects of color were stronger for categories with 
structurally similar characteristics. This is because naming an object requires finer 
within-category differentiation. The finer the required differentiation, the stronger the 
effects of surface detail (Price and Humphries, 1989). 

Wurm, Legge, Isenberg, and Luebker (1993) questioned color's role in object 
recognition using participants who had normal and degraded color vision. They believed 
that color and shape information interacted in object recognition, especially when spatial 
resolution was degraded. They also hypothesized that color would improve object 
recognition for individuals with degraded visual acuity. Previous findings indicated that 
color had no practical advantage for low visual reading, but the lack of research on low 
vision object recognition prevented the correlation between color's involvement in 
reading with object recognition (Wurm, et al., 1993). 

The authors chose food items as objects for viewing. This was because using 
man-made objects, such as those used in prior research, possessed stereotypical shapes. 


Food objects tend to have a wide range of color and shape representation. For example, 
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an apple may be red, green or yellow, and may be whole, sliced or bitten. The authors 
speculated that the reason for the inconsistent findings in previous recognition studies 
was that luminance was not controlled between the chromatic and achromatic imagery. 
The authors eliminated any effect of luminance prior to conducting the experiment. 
Participants were shown 84 images (21 items in four different poses) in one of four 
different formats. These formats were both blurred color and monochrome and unblurred 
color and monochrome. Using a naming task, participants were required to rapidly name 
the food item shown. There were two major findings from the results. First, color did 
improve the recognition of food objects and second, an interaction between color and 
spatial resolution did not exist. This implies that color appears to improve object 
recognition equally for both blurred and unblurred targets (Wurm, et al., 1993). 

Next, the authors tested the vision deficient participants. Each participant was 
categorized according to specific visual impairment. The participants were tasked to 
view each of the 21 food items in four different poses for a total of 84 different 
presentations. The images were magnified by a factor of two to facilitate viewing. Each 
participant equally viewed images in either a color or achromatic representation. The 
authors hypothesized that color would benefit participants with visual deficiencies in 
object recognition (Wurm, et al., 1993). 

Although participant reaction times for identifying images viewed in color was 
half that of images viewed in monochrome, statistical analysis failed to show 
significance. These results were probably due to the large variability of the data due to 
the heterogeneity of the different visual disorders. The authors concluded that color 


enhances object recognition for people with low color vision, but object recognition 1s 
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hindered by poor visual acuity. They also found that color and visual acuity acted 
independently in low vision object recognition (Wurm, et al., 1993). 

Finally, the authors attempted to determine the relationship of color and 
prototypical images. Prototypical images were defined as objects that possessed a typical 
color and shape. Each of the objects was ranked for prototypically. It must be clarified 
that prototypical objects are much different than objects with diagnostic colors. While 
the color red may be prototypically associated with an apple, it is also associated with 
numerous other objects such as tomatoes and cherries. However, orange is associated 
with relatively few objects, namely carrots and pumpkins, that are identified by their 
distinctive profiles. Two factors must be present to determine when color is diagnostic. 
First, the color must be symptomatic of the object (i.e. green is a symptom of spinach), 
and the other objects presented in the allowable domain should not be of this color. 
Therefore, since other foods in the experiment were also green, green was not 
"diagnostic" of spinach (Wurm, et al., 1993). It was expected that object reaction times 
would be smaller for the most prototypical images. This was because prototypical 
images may be recognized faster because they contain certain shape characteristics that 
are stored in memory. Since less prototypical images may have less defined shape 
attributes, it is expected that the advantage of color increased indirectly with prototypical 
images. Participants were asked to estimate probabilities of the food categories given a 
specific color. The results suggested that the participant's knowledge about the color of 
the food item did not account for the advantage of color in object recognition. To 
conclude the experiments findings, the authors found that color improved object 


recognition but the mechanism was sensory rather than cognitive (Wurm, et al., 1993). 
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Joseph and Proffitt (1996) conducted a series of experiments to determine 
semantic versus perceptual influences of color in object recognition. They theorized that 
a significant reason for the contradictory results of previous research was that stored and 
surface color knowledge must play a role in recognition tasks. Surface knowledge is 
defined as the depicted color that represents the surface of the object, such as the color 
yellow for a banana. However, surface color can be arbitrary because an object can be 
represented by any color. Stored color knowledge is the semantic information about the 
prototypical colors of objects, such as the knowledge that bananas are typically yellow. 
Stored color knowledge is not arbitrary because people have shared ideas about the 
typical colors of objects (Joseph and Proffitt, 1996). The authors felt that an objects 
surface color must be compared to the activated prototypical color of the image for 
recognition. The basis for their research was to determine whether the decision that two 
objects were the same depends on the activation of stored color knowledge or on the 
processing of surface color (Joseph and Proffitt, 1996). 

Using a series of experiments in which they tested the effects of surface color; the 
effects of stored knowledge; and the effects of combined surface and stored knowledge in 
verification tasks, they found that congruent color knowledge (red apples) made 
verification easier than incongruent (purple apples) surface color. Although these results 
conflicted with Biederman and Ju (1988) findings, they contributed these results to 
objects used in the imagery. It was assumed that natural objects have stronger 
associations with colors than artifactual objects. Therefore, the use of surface color as a 
cue was more beneficial for object recognition from natural categories (Joseph and 


Proffitt, 1996). They also found that stored color knowledge enhanced object 
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recognition. When both image and word stimuli were similar by shape and prototypical 
color, more interference occurred when compared to stimuli that were similar by only 
shape. When they compared stored and surface color information for object recognition, 
they found that stored color knowledge had a more profound influence than surface color 
knowledge (Joseph and Proffitt, 1996). 

Obviously, the effects that contributed to the inconsistent results were primarily 
due to the experimental methodologies used to conduct each task. It appears that color 
has no influence when viewing artifactual images because prototypical effects are not 
pronounced. Additionally, presenting a label prior to an image in verification tasks may 
prime the participant and mask the surface color effects (Joseph and Proffitt, 1996). 
Throughout these research examples, items were presented in singular form and 
displayed against a neutral backdrop in a controlled environment. No such experiment 
compared images displayed in the global environment. Therefore, it cannot be assumed 
that a participant may be able to identify an apple in a tree faster when presented 1n color 
than in monochrome. Knowing that natural color may generally enhance object 
recognition, the same theories that are applied to pseudo-color that is displayed by fused 
sensor displays may be erroneous. Research into exploring the potential benefits of 
adding color to a scene 1s currently ongoing and also shows inconsistent results. The 
following is a brief synopsis of some research studies being conducted concerning color 
fusion. 

Current sensor fusion research that examines the potential advantages of pseudo- 
color over monochrome is based on three types of experiments. These types are paired 


comparisons, reaction time and mean detection accuracy, and the accuracy of target 
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detection. 

In the paired comparison type of experiment, participants are presented a random 
sequence of image pairs and are asked to respond to the "better" presentation. The goal 
of this type of experiment is to show differences in the preferred ordering of the sensors 
and to deduce if fusion based sensor imagery is more preferred than an individual sensor 
display. This type of experiment was utilized by Krebs, Buttrey, Lewis and McKenzie 
(1997) to identify a preferred sensor for use in a low-vision environment. In this 
experiment, participants were shown pair-wise comparisons of 25 different scenes using 
five different sensor formats. These formats were image intensified (I ), infra-red, fused 
monochrome and two different fused color algorithms. The participant was tasked to 
choose a preferred format while locating a navigational cue contained within the scene. 
The results of the experiment were inconclusive. It was assumed that the inconsistencies 
that resulted from the experiment were due to a sensor to scene interaction. Sensors 
appeared to perform differently depending on the presented scene characteristics such as 
the texture, color and content. Although a slight advantage in recognizing objects using 
the color-fused techniques was shown, participant performance was actually very 
dependent on the type of navigational cues used. Results across all conditions were 
inconsistent (Krebs, Buttrey, Lewis and McKenzie, 1997). 

Measurement of search accuracy and reaction times appear to provide a better 
foundation in the effort to determine if sensor fusion imagery is preferred over 
achromatic imagery. Steele and Perconti (1997) conducted a series of tests to determine 
if the use of color fusion imagery enhanced the performance or improved the situational 


awareness of helicopter pilots during low-visibility flights. The experiment was a part- 
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time simulation and identification task using 25 different scenes collected from the sensor 
combinations in both still frame and video format. The Advanced Helicopter Pilotage 
System (AHPS) was used to collect a series of night pilotage maneuvers using various 
sensor combinations. This imagery was then used to determine horizon perception, 
object recognition, identification and geometric perspective tasks. In the still-frame 
experiment, scenes were randomly presented for at most 10 seconds. Participant 
response time (YES/NO) and accuracy data was collected from specific information 
extracted from the scene. Object recognition and identification tasks required the 
participants to determine if a specific object was present or not. These tasks also required 
the participant to locate a specific object and determine its position in the field of view or 
to describe the object in detail. An observer was also required to determine if any object 
existed that would hinder the flight while the aircraft was on its intended flight path. 
Horizon perception tasks required participants to determine whether the horizon was 
level or not. The geometric perspective tasks required the participant to identify the 
shape or orientation of a feature using monocular perception cues (Steele and Perconti, 
1997). The results of the still-frame experiments showed that fused imagery, whether 
color or monochrome, led to better performance, but the performance was highly 
dependent on the particular fusion algorithm, the particular visual task, and the scene 
content (Steele and Perconti, 1997). 

The second experiment utilized short (17 to 31 second) video clips that were 
collected by the AHPS. These video clips were presented to participants in one of five 
different formats that tested response time and accuracy while identifying a specific 


object. Following each video sequence, participants were asked to rate the imagery based 
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on the five point Likert scale. The results failed to identify a preferred individual sensor 
format. The preference of a sensor format was scene specific (sensor to scene 
interaction) and that no particular sensor was found to be "desirable" for night helicopter 
pilotage. This experiment confirmed the previous experimental findings that sensor 
fusion is highly dependent on the algorithm used, as well as task and scene content 
(Steele and Perconti, 1997). 

Toet, Ijspeert, Waxman and Aquilar (1997) conducted a study to investigate 
whether increasing scene detail with sensor fusion would improve a participants ability to 
perform a situational awareness search task. Images were collected using both visible 
and infra-red sensors that consisted of scenes taken at dawn when low light conditions 
and minimal thermal contrast existed. During the image collection process, a target was 
presented that contained a high infra-red contrast. The images were fused into four 
different formats consisting of color and monochrome. Participants were instructed to 
identify location and error distances. The data for the incorrect responses was then 
calculated. The results showed that participants performed better using fused color 
images when compared to fused monochrome and individual sensor images (Toet, et al., 
1997): 

These inconsistent sensor fusion studies may be attributed to the differences in 
fusion algorithms used in the various experiments and to the differences in the employed 
psychophysical tasks. A reason for the inconsistent results may result because fused 
imagery does not always improve participant performance when compared to individual 
sensor imagery. There may exist certain conditions when the fusion process aids 


performance and other situations when fusion is not necessary. If this were true, the 
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experiments should be designed that precisely elucidate when and where the fusion 
process is helpful. A more probable explanation and one that is easy to control is that 
proper experimental methodology for adequately testing these images has not yet been 
employed. These previous studies contain potential experimental methodological 
problems that may hinder accurate evaluation of the various image processes. 


G. HYPOTHESIS 


A significant deficiency in color fusion research is that researchers have 
attempted to utilize the findings from natural and achromatic studies and correlate the 
principles to the false-color theories that are used in sensor fusion. The majority of the 
color studies were conducted in the laboratory environment, utilizing single, defined 
natural or manmade objects observed in an unnatural setting. Comparing the findings 
from these types of experiments to research being conducted with natural scenes and a 
mixture of living and non-living objects is a significant threat to external validity. No 
known studies have been found that compare HVS perception of a global scene with the 
same scene observed with imagery obtained from a fused color sensor. 

The experimental complexity of sensor fusion studies has proven to be 
overwhelming in the search for verification that it provides a better, more appealing scene 
for the observer. Efforts have generally focused on the end product of object recognition 
and have only shown that the performance of fusing combinations of night sensors varies 
with the algorithms used and scene condition. A significant deficiency to sensor fusion is 
that in some environmental conditions, observers have exhibited better performance with 
individual sensor imagery. The research conducted espouses the superiority of fused 
imagery by using specific environmental conditions or subjective operator opinion. In 


light of these inconsistencies, this thesis will attempt to relate individual preferences 
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between identical scenes manipulated with various color, pseudo-color and achromatic 
formats. 

Natural scenes have been carefully selected by pre-designated standards taken 
from a commercially available collection of digital photographs on a Softkey Photo 
Gallery CD published by the Learning Company. Some of the categories of images were 
scenes from the states of Washington, Utah, and Colorado; farming scenes, and pictures 
of animals; and pictures of sporting events. Each sensor was identically manipulated so 
that the only difference a participant would see was the particular color or 
monochromatic format. Reducing the variability of processing with same scene displays 
controls the cognitive process. The feature maps associated with an image and the effects 
of sensor ficion can then be determined. The feature maps are dependent on the overall 
scene context. Sufficient scenes have been selected to ensure the experiments look across 
multiple real world scene environments and not just those that have shown superior 
fusion performance. 

Two experiments were conducted to measure subject accuracy (error rates in 
percent) and reaction time (in milliseconds) by determining the orientation of natural 
color, artificial color, and achromatic scenes. The purpose of the experiment is to 
determine which type of scene is preferred between natural color, false color or 
monochrome, and to identify specific deficiencies that may cause false color scenes to be 
processed less efficiently. 

The second experiment will be identical to the first experiment except Gaussian 
noise will be substituted for false saturation. This experiment will compare the findings 


of the previous experiment under ideal conditions with the findings under conditions of 
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low resolution. This may add validity to previous findings that color is useful to help 
segment object shapes in the absence of defined edge information. Additionally, this 
experiment will closely match true representations of imagery displayed to a user when 
using a fused device under realistic conditions. 

In lieu of the previous studies, it is expected that reaction times and accuracy for 
false color scenes will be slower and less accurate when compared to a natural color or 
achromatic scene. Under degraded conditions, it is hypothesized that the effect will be 
more severe, that natural color imagery will be highly preferred compared to a false-color 
and achromatic scene. A level of significance of @= 0.5 will be used for all statistical 


purposes. 
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Ill. EXPERIMENT I 
A. METHODS 

1. Participants _ 

Twenty students from various military services and job specialties undergoing 
academic studies at the Naval Postgraduate School or at the Defense Language Institute 
volunteered for this study. All participants, by self-report, possessed normal color vision 
and had at least 20/20 corrected vision. Participants were uninformed concerning the 
purpose of the experiment. Before participating in the experiment, participants were 


granted informed consent. 


2. Apparatus 


The experimental workstation consisted of a 200 MHZ Pentium personal 
computer equipped with a Texas Instrument TMS-340 Video Board and _ the 
corresponding TIGA Interface to Vision Research Graphics software. Stimuli were 
presented on an IDEK MF-8521 High Resolution color monitor (21” X 20” viewable 
area) equipped with an anti-reflect, non-glare, P-22 short persistence CRT. Pixel size 
was .26” horizontal and .28” vertical; the resolution was 800 X 600 square pixels and the 
frame rate was 98.7 Hz. The brightness of the monitor was linearized by means of an 
eight-bit color look-up table (LUT) for the red, blue and green guns. Responses were 
recorded on the number pad of a standard (IBM clone) keyboard. Moderate ambient 
luminance (1.2 cd/m* luminance) was maintained using a small floor lamp placed on the 
floor behind the IDEK monitor. Viewing distance was approximately 150 centimeters 
and the subjects were free to move their heads. A chair was provided for subject comfort 
and to help maintain the appropriate distance and viewing angle. Overhead lights were 


extinguished and subjects were required to adapt to the darkness for approximately two 
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minutes prior to conducting the experiment. 


3: Stimuli 


Each of 270 images (30 of which were designated for practice) was presented in 
one of eight different formats. These formats were natural color, color with false hue, 
saturation, or both hue and saturation manipulated, natural gray and gray with false hue, 
saturation, or both hue and saturation manipulated. Images were also presented in an 
upright or inverted position, so that each image was presented in 16 possible ways. 

Construction of the experiment stimuli began by choosing images under specific 
pre-arranged criteria determined by two vision science experts. The criterion for 
selection was that a horizon was not present to contribute to identifying orientation. 
Images that contained reflections and shadows were eliminated because they may 
confuse the participant. All image manipulations were performed using Adobe 
Photoshop Version 4.0 Illustrator Software. The images were first cropped to a square 
400 X 400 pixel size. Each of the 270 images was then manipulated within the 
Photoshop software using the standard image processing functions. 

The hue of the images was manipulated by adjusting the hue value of each pixel 
and replacing it by its complementary color. The concept of complementary colors arises 
from a modern version of Newton’s color wheel. The color wheel organizes all the colors 
according to the visible light spectrum wavelengths. The exception to the color wheel 
organization is the region that is complementary to the green region. This region is called 
the nonspectral hue. It is produced by various combinations of the other colors. The 
color magenta is a combination of the blue and red colors (Figure 10). The remainder of 
the colors on the color wheel represent only the monochromatic colors — those colors that 


are produced by a single wavelength (Matlin and Foley, 1997). 
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The complement of a color is found by drawing a straight line from a particular 
color, through the wheel center, to the intersection on the opposite side of the wheel. For 
example from Figure 10, the complementary color of yellow is blue. It was in this 
manner that the color of the images in the two experiments were changed. 

There are two different ways in which color is mixed. Additive mixture of colors 
infers that particular light beams are combined, not the color pigments from the 
respective parts of the spectrum. Additive mixing of complementary colors, such as 
mixing blue and yellow in equal portions, produces gray. The subtractive mixture of 
colors infers that the color pigments are mixed, or that two or more colored filters are 
placed together and a light beam is emitted through these filters. The subtractive mixing 


of blue and yellow produces green (Matlin and Foley, 1997). 


Yellow 





Figure 10: Ulustration of the modern color wheel. 
Courtesy of http://www.ricklineback.com/term4.htm 


The Photoshop software provides a slide bar with a range from —180 degrees 


(counterclockwise from 0) to +180 degrees (clockwise from 0) for hue manipulation. 
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Rotating along the color wheel —180 or +180 degrees results in the exact same outcome; 
the original image’s color complement. For consistency, all images were reassigned a 
value of +180 degrees. Figure 11 shows an original image and the image's color 
complement (image manipulate for hue). As seen, altering an image's hue gives it an 


unnatural appearance. 
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Figure 11: Natural colored image and image manipulated for Hue. 
Courtesy of Softkey Photo Gallery CD published by the Learning Company 


The image's saturation was manipulated by taking each color pixel and fully 
saturating the color. Colors high in purity (saturation) are arranged around the edge of 
the color wheel. As the saturation level moves toward the center of the color wheel, the 
colors become increasingly less pure and more faded, moving toward an evenly balanced 
mixture of wavelengths with no single wavelength dominance (Sekular and Blake, 1994). 
Referring to the color wheel in Figure 12, saturation is seen as the apparent purity of a 


color. 
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Figure 12: Color wheel plus Saturation. Notice how the 
color blue becomes lighter as it moves toward the center of 
the color wheel. (Matlin and Foley, 1997) 


Photoshop software provides a slide bar with a range from —100 to +100. An 





image reassigned a value of +100 is considered to be totally saturated. An image 
reassigned a value of -100 is considered to be achromatic. All images were reassigned a 
value of +100. Figure 13 shows an original image and the image manipulated by 
changing its saturation. For example, a saturated blue, one lying on the color wheel edge, 
is perceived as deep hue; an unsaturated blue, close to the color wheel origin at the same 
hue, is perceived as a very light blue or baby blue. As seen, altering an image's saturation 


will not adversely change the color, it only changes the brightness of the image. 
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Figure 13: Natural colored image and image manipulated for Saturation. 
Courtesy of Softkey Photo Gallery CD published by the Learning Company 


To manipulate an image for both hue and saturation, the values for the images 
were reassigned by the Photoshop software a hue value of +180 degrees and the 
saturation value of +100. Figure 14 provides a visual representation of the original image 


and the image with both the hue and saturation manipulated. 
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Figure 14: Natural colored image and image manipulated for Hue and Saturation. 
Courtesy of Softkey Photo Gallery CD published by the Learning Company 


Achromatic images were first manipulated in the same fashion as the natural 
colored images, then converted to grayscale. The purpose of the grayscale images was to 
provide a control for changes in luminance that might accompany manipulations of color. 


Each image representation 1s seen 1n Figure 15. 
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Figure 15: Achromatic images: A) natural gray, B) gray with Hue manipulated, C) 
gray with Saturation manipulated, D) gray with Hue and Saturation manipulated. 
Courtesy of Softkey Photo Gallery CD published by the Learning Company 


4. Procedure 


To prevent learning, each subject participated in one but not both experiments. 
Additionally, each image was viewed once in one of the various formats. Each subject 
was thoroughly briefed only on the background and procedures of the experiment and 
was given the opportunity to ask questions. Subjects were instructed to indicate as 
rapidly and accurately as possible with A keypress, whether the image orientation was 
upright (by pressing 1) or inverted (by pressing 2). Each image remained on the screen 
until the subject responded. A computer generated audio tone provided feedback 


following each incorrect response. Subsequent images were presented following a 
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response after a slight delay. After 30 consecutive image presentations, the experiment 
paused to allow the subject an opportunity to relax. The experiment restarted when the 
subject pressed any key. The entire experiment lasted approximately 20 minutes. 

Each subject was given a block of thirty practice trials prior to the experiment. 
The practice trials were presented in the same format as the experiment. Upon 
completion, subjects were offered a brief pause to ask any questions. The actual 
experiment then commenced and the participants were presented with 240 images. 
Reaction times and accuracy were collected for each trial. 

5. Experimental Design 

The design was a 2 X 2 X 2 X 2 within-subjects factorial design. Factors 
included orientation (upright or inverted), color (color or gray), hue (natural or false) and 
saturation (natural or false). Each of the 16 cells contained 15 observations for a total of 
240 data points. Reaction times for incorrect responses and responses to the 30 practice 
images were not included in the data analysis. There were no theoretical expectations of 
the colors effect on inverted images from prior research, therefore, the reaction times to 
inverted images were also discarded. However, inverted error rates were recorded and 
viewed to ensure participants did not display any bias toward a particular orientation 
selection. 


B. DEVELOPMENT OF THE VISUAL MODEL 


As with any psychophysical experiment, inconsistencies occur between each 
subject’s response times and error rates. These differences arise in part due to variances 
in individual participants, image conditions and manipulation. To account for these 


differences, the following model is proposed: 
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Reaction Time = Part, + Color , + Hue, + Sat, + Image,, + error, 


ikimn 


where Part, represents the ith participant of the experiment; 

Color , represents the color of the image (natural or monochrome); 

Hue, represents the hue condition of the image (natural or false); 

Sat, t represents the saturation condition of the image (natural or false); 

Image ,, represents each presented image and 

Error, represents for unaccounted variations within the model. 

Oe RESULTS 

To determine the appropriate statistical method for the results of the experiment, 
normality tests were conducted and showed that the data failed to follow the properties of 
normality. Therefore, the assumption that the original data was normal was not tenable. 
As seen below, the analysis will proceed on intra-subject differences in reaction times 
and error rates and while these might not be exactly normal, the t-test is well-known to be 
robust against minor deviations from normality. 

Mean reaction times (in milliseconds) and mean error rates (in percent) for each 
of the participants (Appendix C) were calculated from individual performances for each 
condition. To ensure participants did not show any bias toward any particular keypress, 
the mean error rates were observed for the upright and inverted conditions. If the error 
rates for the upright representations were zero percent, it would indicate that the 
participant displayed a bias towards the "1" keypress. If this were the case, the mean 
error rates for the inverted orientations would be very large. For this experiment, no bias 
toward a particular keypress was noted. Additionally, comparing the mean reaction time 


and error rates for each condition may show the existence of a speed/accuracy trade-off. 
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A low error rate and unusually large reaction-time indicates that a participant is very 
careful in selecting the proper orientation. Conversely, a high error rate and small 
reaction time indicates that a participant is hasty in making decisions. The experimental 
data correlates with respective categorical speed/accuracy trade-off relationships in that 
higher reaction times were generally accompanied by higher error rates. When images 
were manipulated from their natural color or gray conditions, participant reaction time 
and error rates increased. Box plots and line plots of each condition are seen in Figures 
16 and 17. The box plot was chosen because it gives readers quick visual information 
about the data spread within a variable and offers a comparison between variables. The 
box plot consists of a few basic elements: the upper (shaded area) quartile, the lower 
(shaded area) quartile and the median (the white line in the middle). The whiskers show 
the interval of values outside the box, or values that are no more than one and a half times 
the interquartile range. The interquartile range is defined as the difference between the 
upper and lower quartile. The horizontal lines far outside the boxplot represent outliers 


(Hamilton, 1992). The tabulated data can be seen in Table 1. 


Box Plotof Mean Reaction Times for all Conditions 
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Figure 16: Box Plot of all image condition mean reaction times 
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By inspection of the box plot (Figure 16), the data for the natural color condition 
shows a lower median reaction time and smaller variance than the other image 
conditions. Outliers are shown to exist for each condition. These outliers are attributed 


to two specific participant's performance, implying that these participants were 


consistently slow across each condition in relation to other participants. 
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Figure 17: Line Plots of mean reaction times and error rates for Color and Gray 
conditions. 
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The line plot (Figure 17) offers a pictorial representation of Table 1. Generally, 
the mean error rates and reaction times are less for color conditions than for achromatic 
conditions. The reaction times for the color false hue and gray false hue conditions do 
not follow this trend, thereby displaying a possible effect for the color false hue 
condition. Also, the error rates between the color false hue and saturation and gray false 
hue and saturation condition appear to be fairly consistent between conditions, implying 
that a speed/accuracy trade-off does not account for these results. The remaining error 
rates and corresponding eee time increased for images that were viewed in 
achromatic formats. 

Table 1 shows that the average reaction times for the natural color and natural 
gray condition are smaller than the other image conditions. Smaller reaction times for 
these conditions were expected by the developed hypothesis because true scene 
representation is easier to detect than scenes that do not symbolize the true representation 
of the scene. Additionally, the error rate for natural color imagery was smaller, 
indicating that participants who view images under natural color conditions will not only 
respond faster but will be more accurate in object recognition. Images that were 
manipulated for hue and saturation produce longer reaction times and error rates, thereby 
supporting the hypothesis that manipulating the color of an image will disrupt the 
participant's ability to efficiently process image features. 

GRAY 


CFS CFHS NG GFH GFS GFHS - 
664.76 726.20 | 656.52 673.75 680.50 745.52 


34.05 63.13 S177 62.55 ees. 13 386m 60-20 (74.94 
5267 2066) 7.00 8.67 G35 8.67 6.67 13.00 
1.02 1.33 LS Le) 1.64 1.82 153 2.49 
Table 1: Tabulated data for each conditions mean reaction time and error rate. 
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where RT represents the reaction time of the participant selecting an orientation; 
SEM represents the standard error of the means; 
NC represents the natural color condition; 
CFH represents the color false hue condition; 
CFS represents the color false saturation condition; 
CFHS represents the color false hue and saturation condition; 
NG represents the natural gray condition; 
GFH represents the gray false hue condition; 
GFS represents the gray false saturation condition and 
GFSH represents the gray false hue and saturation condition. 

Because we hypothesize that there is a difference in reaction time between natural 
and artificial color scenes, the difference between each condition's mean and natural 
color's mean should be positive. Table 2 shows the difference of the means between each 
condition. 


Color Gray 
DIFF CFH-NC CFS-NC CHFS-NC! GFH-NG GFS-NG GFHS-NG 


138.98 63.71 125.14 17.24 aS io 89.00 


34.80 34.80 35.60 38.40 47.59 53.08 
1.67 3.33 5.00 2.33 0.33 6.67 
1.59 1.49 1.67 2.07 2.08 2050 
Table 2: Difference of the reaction time and error rate means across all 
conditions. Notice all values are positive indicating that reaction time 
and error rates for natural color scenes are smaller. 





The difference of the means of each condition in Table 2 is not the same; 
therefore, we may conclude by inspection that there can be an effect on reaction times for 
each condition. These positive values indicate that the participant, when observing 


natural conditions will exhibit better performance. The comparison results displayed in 
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Table 2 strengthen the assumption that individual perception is negatively disrupted when 
altering an images natural condition. To test the data, a paired t-test using Bonferroni's 
inequalities with an @= 0.05, six samples and a confidence level of 99.2% was used. 
When comparing the color false hue condition versus the natural color condition reaction 
time (t(19) = 3.9935 p < .008) and accuracy (t(19) = 1.045 p = .008), an effect for 
reaction time was found. This indicates that a participant's perception is disrupted when 
changing the image’s hue. Comparing the color false hue and saturation versus the 
natural color condition reaction time (t(19) = 3.5148 p < .008) and accuracy (t(19) = 
3.0009 p < .008), an effect was also found. Not only were the participant reaction times 
significant, but the error rates were also significant. The fact that the combined hue and 
saturation condition was significant-‘may mean that an interaction between the conditions 
might exist. A test for the presence of interaction will be conducted later. The remaining 
comparisons from Table 2, color false saturation versus natural color condition reaction 
time (t(19) = 1.8308 p = .008) and accuracy (t(19) = 2.2361 p 2 .008), gray false hue 
versus natural gray condition reaction times (t(19) = t = 0.4489 p 2 .008) and accuracy 
(t(19) = 1.1278 p = .008), gray false saturation versus natural gray condition reaction 
time (t(19) = 0.5041 p = .008) and accuracy (t(19) = 0.1599 p = .008), and gray false 
hue and saturation versus natural gray condition reaction time (t(19) = 1.6768 p 2 .008) 
and accuracy (t(19) = 2.8133 p = .008) revealed no effect. The absence of an effect for 
color false hue versus natural gray provides evidence that manipulating an image by 
altering the saturation may not effect a participant's reaction time. This also applies to 
images viewed under monochrome conditions where manipulation for saturation did not 


affect a participant's perception. These results may add evidence that when altering an 
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image by hue and saturation, the hue effect may influence the combined hue and 
Saturation effect on participant reaction time. Since achromatic images contain neither 
hue nor saturation levels, the Statistical t-tests agree with the inference that there is no 
significant difference between the natural gray and false gray conditions. 

Luminance tests were also conducted by comparing the false gray conditions to 
the natural gray images. For luminance to be a factor, the difference of the means 
between false gray and natural gray conditions shall be a value other than zero. Table 2 
shows that these differences are positive and the values are much less than the values of 
the color conditions. However, as noted above, respective statistical t-tests showed no 
effect across each false gray condition when compared to the natural gray condition. 
Therefore, the effects of the color conditions were due to color and not to a luminance 
confound. 

To determine if altering an image's hue had any effect on participant reaction time 
and error rates, the hue effect was removed from the images by subtracting all conditions 
of natural hue from the corresponding conditions of false hue. For example, the 
difference in the mean reaction time and error rates for the conditions of color false 
saturation and both color false hue and saturation removes the saturation effect and leaves 


only the effect of hue on the image. Under the hypothesis, there will be a difference in 


the reaction times of these conditions. Table 3 displays the results. 


ay) 


HUE 
DIFF CFH-NC CFHS-CFS GFH-NG  GFHS-FGS 
Mean RT 138.98 61.43 17.24 65.01 


SEM 34.80 43.23 38.40 42.03 

Mean Err 1eG7 5.00 2h 3 6.67 

SEM 1.59 evi Zn 23) 

Table 3: Difference in reaction time and error rate means across 
all image conditions with and without Hue. 





To test the data, a paired t-test using Bonferroni's inequalities with an a@= 0.05, 
four samples and a confidence level of 98.75% was used. In tests of the color false hue 
condition versus the natural color condition reaction time (t(19) = 3.9935 p < .0125) and 
accuracy (t(19) = 1.045, p 2.0125), a reaction time effect was found. This effect has 
already been explained above. The remaining comparisons from Table 3, color false hue 
and saturation versus color false saturation condition reaction time (t(19) = 1.4212 p 
2.0125) and accuracy (t(19) = 0.7373 p 2.0125), gray false hue versus natural gray 
condition reaction time (t(19) = 0.4489 p> .0125) and accuracy (t(19) = 1.1278 p2 
.0125), and gray false hue and saturation versus gray false saturation condition reaction 
time (t(19) = 1.5467 p> .0125) and accuracy (t(19) = 2.2637p2> .0125) showed no 
eitect: 

To determine if altering an image's saturation had an effect on participant reaction 
time and error rates, the saturation effect was removed from the images by subtracting all 
conditions of natural saturation from the corresponding conditions of false saturation. 
For example, the difference in the mean reaction time and error rates for the conditions of 
color false hue and both color false hue and saturation removes the hue effect and leaves 
only the effect of saturation on the image. Under the hypothesis, there will be a difference 


in the reaction times of these conditions. Table 4 displays the results. 
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SATURATION 
CFS-NC | CFHS-CFH  GFS-NG  GFHS-FGH 
Gan -13.83 23299 


34.80 24.93 47.59 
3.33 3.33 0.33 
1.49 ~ 1.64 2.08 
Table 4: Difference in reaction time and error rate means across all 
image conditions with and without Saturation. 





To test the data, a paired t-test using Bonferroni's inequalities with an @= 0.05, 
four samples and a confidence level of 98.75% was used. In tests of the color false 
saturation condition versus the natural color condition reaction time (t(19) = 1.8308 p = 
0125) and accuracy (t(19) = 2.2361 p 2 .0125), color false hue and saturation versus 
color false hue condition reaction time (t(19) = -.5549 p = .0125) and accuracy (t(19) = 
2.0329 p 2 .0125), gray false saturation versus natural gray condition reaction time (t(19) 
= .5041 p 2 .0125) and accuracy (t(19) = 0.1599 p = .0125), and gray false hue and 
saturation versus gray false hue condition (t(19) = 2.2254 p = .0125) and accuracy (t(19) 
= 2.0409 p 2 .0125), failed to produce evidence that altering an image's saturation effects 
participant reaction and error rates. Each condition indicates that whether viewing a 
natural color image or achromatic image, an effect on participant reaction time and error 
rates when viewing an image that has been altered for saturation does not exist. This is 
because changing the image saturation does not drastically change the image color, it 
only brightens the entire color spectrum of the image. It is interesting to note that 
participant reaction time when viewing a false color image that had been manipulated for 
hue and saturation was less than the reaction time when viewing a color image that had 
been manipulated for hue. This may suggest that the brightness of the image due to the 
altered saturation level may enhance participant performance when compared to a color 


image that was manipulated for hue. 
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To determine the effects of color, we compare the differences between the mean 
reaction time and error rates of each color condition with the means of the same images 
in the achromatic condition. For example, the difference in the reaction time and error 
rate means between natural gray and the natural color condition will be negligible. Fora 
condition of color to be a factor, there will be a difference in the means across each 
condition. Table 5 displays the results. 


GRAY-COLOR 
NG-NC  GFH-CFH GFS-CFS GFHS-CFHS 


55.47 -66.28 15.74 


31.65 25074 45.44 
2.67 3.33 -0.33 
1.63 1.71 28 
Table 5: Difference in reaction time and error rate means across 
all image conditions between Color and Monochrome. 





To test the data, a paired t-test using Bonferroni's inequalities with an a@= 0.05, 
four samples and a confidence level of 98.75% was used. When comparing gray false 
hue versus color false hue condition reaction time (t(19) = -2.7912 p < .Q125) and 
accuracy (t(19) = 1.633 p = .0125), a reaction time effect is found. As seen in Table 5, 
the reaction time for the gray false hue image is much smaller than the reaction time for 
the color false hue image. This indicates that participants are more apt to perceive an 
achromatic image faster than a color image that is altered for hue. This provides more 
evidence that the effect of hue is significant for a colored image. The other conditions, 
natural gray versus natural color condition reaction time (t(19) = 1.7524 p 2 .0125) and 
accuracy (t(19) = 1.949 p = .0125), gray false saturation versus color false saturation 
condition reaction time (t(19) = .3465 p = .0125) and accuracy (t(19) = -0.1563 p 2 
0125), and gray false hue, saturation versus color false hue, saturation condition reaction 


time (t(19) = .5086 p = .0125) and accuracy (t(19) = 1.8183 p = .0125), show no effect. 
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To employ a second method to validate the visual model, a statistically reliable 
one-sided t-test was conducted on the combination of similar image conditions using an 
a= 0.5. By grouping individual conditions of a particular group (Figure 18) and 
comparing these conditions reaction time and error rate means that do not possess the 


groups stated manipulation, the following result is shown in Table 6. 


COMPARING HUES 


Color False Hue Natural Color 
Color False Hue Color False 
and Saturation Saturation 


a 
ee 

Gray False Hue 5 Natural Gray 
ae ae 


Gray False Hue Gray False 
and Saturation Saturation 
Figure 18: Stacked comparison of all viewing conditions 
with and without Hue manipulated 


Difference between Natural Hue 
and false Hue conditions 
Mean RT 


SEM 
Mean Err 
SEM 













Table 6: Reaction time and error 
rate means’ for _ images 
manipulated for Hue 


Based on the data provided in Table 6, there is evidence that a reaction time (t(79) 
= 3.8807 p < .05) and accuracy (t(79) = 2.6982 p = .05) effect exists. This reaction time 
effect 1s also present in the previous tests that compare hue and images that are not 
manipulated for hue. These results provide evidence that manipulating an image's hue 
will disrupt participant visual perception. 


To determine if a saturation effect exists, reaction time and error rate means of the 
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images that contain natural saturation were compared to the reaction time and error rate 
means of the images that contained false saturation. Figure 19 shows these conditions. 


COMPARING SATURATION 


Saturation 
and Saturation 


Gray False Natural Gray 
Saturation 
Gray False Hue 
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Gray False Hue 
and Saturation 


Figure 19: Stacked comparison of all viewing conditions 
with and without Saturation manipulated 


Difference between Natural 
Saturation and false non- 
Saturation conditions 





Table 7: Reaction time and error 
rate means’ for images 
manipulated for Saturation 


The difference of the mean reaction times between saturation conditions reaction 
time (t(79) = 2.0236 p < .05) and accuracy (t(79) = 3.0643 p < .05) shows an effect. 
This result contradicts earlier results that test the effect of saturation on participant visual 
perception. The reason for this contradiction may be due to the large standard error for 
the mean reaction time and the increased degrees of freedom that occur by combining 
image conditions. However, it must be noted that the p-value (p = 0.0464) suggests that 
the effect of saturation is not very strong. There may also be some interaction between 
hue and saturation that may enhance the effect of saturation. A statistical test for 


interaction will be performed later. 
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Lastly, a comparison of color to monochrome conditions was conducted to 
determine if a color effect exists. Although altering images by changing color (hue and 
Saturation) was included with natural color conditions, this comparison will determine 
whether color is more preferred than achromatic conditions. Figure 20 shows these 


conditions. 


COMPARING COLOR TO GRAY 


Natural Color a = Natural Gray 


Color False Hue ; i Gray False Hue 


Gray False 


Color False 
Saturation 


Saturation 





Gray False Hue 
and Saturation ——— and Saturation 
Figure 20: Stacked comparison of all viewing conditions 
with and without Color 


Difference between Color and 
achromatic conditions 






Table 8: Reaction time and error 
rate means for images with and 
without Color 


Observing the Table 8 data for comparing color conditions with achromatic 
conditions reaction time (t(79) = -.3337 p = .05) and accuracy (t(79) = 2.5131 p < .05) 
an effect was found. The negative value for the mean reaction times indicates that 
participant reaction times are slightly smaller when viewing images in achromatic format 
when compared to a color format. 


Strong evidence has already been shown that supports the fact that altering an 
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image's hue has a pronounced effect on participant's perceptual abilities. However, 
previous results indicate that an interaction may exist between hue and saturation when 
images are altered by these combined conditions. Using the hypothesis that there exists 
interaction between conditions of hue and saturation, the following linear model is 
developed: 


Reaction Time = Part, + Color , + (Hue, * Sat,) + Image,, + error, 


ijkimn 


where Part, represents the ith participant of the experiment; 

Color , represents the color of the image (natural or monochrome); 
(Hue , *Sat,) represents the interaction between the two variables; 
Image ,, represents each presented image and 

Error ,, represents for unaccounted variations within the model. 


By combining the effects of hue and saturation and running the linear model 
to test for interaction between the variables, no interaction is shown to exist (t(136) = 
-0.3968 p = .05). The individual values for hue (t(136) = -2.3837 p < .05) and 
saturation (t(136) = -1.0921 p = .05) also confirm the effect that altering hue will 
disrupt participant reaction time and error rates. This interaction model is strongly 
supported by a high coefficient of determination (R* ) of 81 percent, which shows the 
amount of the total variance explained by the model. Therefore, it is concluded that 
the effects for hue have a significant effect on participant reaction time. When an 
image is manipulated for both hue and saturation, they act independently of each 
other and it is the effect of hue that causes the increase in participant reaction time 


and error rates to increase. 
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D. DISCUSSION 


The purpose of this experiment is to clarify the role of color in object recognition. 
Previous research has been inconclusive in adequately defining colors role. However, 
equipped with a large abundance of professionally photographed images, this experiment 
makes it possible to manipulate the images in eight different conditions for comparison. 
It was hypothesized that natural color scenes would be interpreted in a similar manner to 
scenes displayed achromatically. However, an artificial color scene altered for hue, 
saturation, or both hue cal saturation, would degrade a participant's visual perception 
process when compared to either natural color or natural gray. 

The methodology for this experiment originates by attempting to clarify colors 
role in the "real-world." Previous research defined colors role in controlled laboratory 
environments without using natural scenes, and defined color as a combination of natural 
color and artificial color. Natural objects were defined as living, and unnatural objects 
defined as non-living. Object recognition failed to emphasize the role of color in global 
scenes. 

This experiment varied the chromatic format of the entire scene. Not only were 
individual objects altered in color, but the background and foregrounds were altered as 
well. Images were defined as natural if they were unaltered and artificial if they were 
altered in hue, saturation, or both. To provide a control measure, achromatic 
representations were developed for each color format. Additionally, luminance effects 
were determined by the comparison of natural gray to the other manipulated = 
conditions. The experimental design employed a forced choice task, where observers 
were to indicate scene orientation. 


Perhaps the preferred method in discussing the results of Experiment 1 is to 
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analyze the data from the combined or stacked image comparisons, then to analyze the 
data from individual image comparisons. This type of discussion will first show that 
generalizing color as both natural and artificial will display results that are different when 
color is analyzed in the respective individual natural and artificial format. 

The results of analyzing the combined comparisons between color and achromatic 
scenes produced an effect for the error rate means. Although the reaction times means 
were Slightly less for achromatic images, participants error more frequently which 
significantly affected their performance. This result suggests that color is beneficial in 
object recognition and validates the majority of previous research. 

It is wrong to suggest that these findings be applied to studies in which artificial 
color is tested. The fact that previous research suggests that the HVS is disrupted when 
viewing non-prototypical objects, applying generalized color findings to experiments that 
alter color, is misleading. The HVS develops color mappings that may be recalled when 
necessary. When a conflict occurs between the recognition of a particular object and its 
stored color mapping features, a disruption occurs in the participant's ability to efficiently 
recognize the objects. The results of the comparisons for images that are altered for hue 
and those that are not manipulated for hue are significant. This finding suggests that once 
a natural scene is altered by changing the color, the HVS is disrupted. Analysis of the 
images that were altered for saturation was also significant. This result 1s surprising 
because when the saturation of the natural scene is modified, the brightness and not 
necessarily the color, is changed. However, it is shown that when an image is altered for 
hue and saturation, the effect for hue overwhelms and effect for saturation and causes the 


combined hue and saturation conditions to be significant, implying that either or both the 
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hue and saturation are disruptive. The results of the statistical tests for saturation are not 
strong and therefore, cannot suggest that image manipulation for saturation significantly 
disrupts the HVS. More studies should be conducted to validate this finding. 

Another statistical method tested the comparisons of each image format against 
combinations of the other formats to find an effect for hue, saturation, and color. Each 
manipulated condition was compared to the natural format, whether it was color or 
monochrome. The differences in the mean reaction times and error rates showed a 
significant effect for color false hue and color false hue and saturation. The non- 
significant results of the achromatic comparison shows that luminance has no effect on 
participant perceptual ability. Because the effect for both hue and saturation is 
significant, the possibility of interaction between hue and saturation must be considered. 
A linear model was developed and results failed to show any hue and saturation 
interaction. Therefore, more evidence exists that suggests altering an image for hue 
disrupts the participant's ability to recognize objects. Image manipulation for saturation 
was then tested. The statistical results show a lack of evidence to suggest any effect on 
participant performance. Image manipulation for hue was then tested. Effects for hue 
were already found in the combined or stacked tests as well as the comparisons to the 
natural formats. The results of hue manipulation shows that an image altered for hue has 
an effect when compared to images in the natural format. Again, this suggests that 
altering the natural color of a scene adversely effects the participants ability to process 
the scene with the same efficiency as in natural color. Lastly, the specific color format 
manipulations were compared to the achromatic conditions. The results showed that 


changing the hue affects the HVS when it is compared to the same scene in achromatic 
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format. One interesting comparison to note is that no effect is found between a natural 
color scene and an achromatic scene. Although only 15 images were compared in these 
formats, this result adds validity to the hypothesis that natural color does not enhance 
object recognition when compared to a natural monochrome scene. 

The results of Experiment 1 showed that depending on the specific experimental 
design employed, different results will occur. Grouping images by color and comparing 
them to images in achromatic format, performance with color imagery was found to be 
better. This result validates numerous color theories concerning object recognition. 
However, when separating the colors into natural and artificial categories, evidence 
shows that color may not be beneficial. As stated, the type of experimental design 
employed reveals different results. Due to the overwhelming evidence that implies that 
altering an images natural color will adversely affect participant performance in object 
recognition, the image format that 1s most preferred is the natural format. If scenes 
cannot be represented in the natural color, then the best alternative is to display the scene 


achromatically. 
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IV. EXPERIMENT 2 


A. METHODS 


Ihe Participants = 
As in Experiment 1, fifteen students voluntarily participated in this study. The 
participants did not take part in the previous experiment. 


Ze Apparatus 


The same experimental workstation used in the previous experiment was used in 
Experiment 2. 

Se Stimuli 

Each of 270 images (30 of which were designated for practice) was presented in 
one of eight different formats. These formats were natural color and gray, and color and 
false gray with hue. Additionally, half of the images in each cell were presented with 
Gaussian noise added. Images were also presented in an upright or inverted position, so 
that each image was presented in 16 possible ways. 

The experiment was constructed by examining the previous experiment stimuli 
and choosing the images that contained salient color contrast as judged by the author and 
one other vision science expert. Objects within the images that encompassed 
approximately 25% or more of the total viewing area were eliminated. These two 
selection processes eliminated approximately half the previous experiment images. 
Replacement images were selected from the same categories of images off the Softkey 
Photo Gallery CD. The images were cropped to a rectangular 250 X 400 pixel size, then 
manipulated using the Photoshop software's standard image processing functions. Image 


manipulations to generate the color false hue, natural gray and gray false hue conditions 
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were identical to those used in Experiment 1. Noise was added by increasing or 
decreasing the assigned value of each pixel by an amount chosen randomly from a 
Gaussian distribution with mean zero and standard deviation of 50. Pixel values of more 
than 255 or less than zero were set to values of 255 or zero respectively. 

Procedures to convert the images to achromatic format were identical to those of 


the previous experiment. Figure 21 depicts each image representation. 
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Figure 21: Image comparison with and without noise. A) Natural Color with Noise, 
B) Color False Hue with Noise, C) Natural Gray with Noise, D) Gray False Hue with 
Noise 


4. Procedure 


The same procedure used for the previous experiment was used in Experiment 2. 
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>: Experimental Design 

The design for Experiment 2 was a2 X 2 X 2 X 2 within-subjects factorial design. 
Factors included orientation (upright or " inverted), color (color or gray), hue (natural or 
false) and noise (noise or no noise). The observations, number of cells, and data 
collection methods were identical to Experimental 1. 


B. DEVELOPMENT OF VISUAL MODEL 


Inconsistencies will also occur between each subject's response time and error 
rates similar to the data from the previous experiment. To account for the differences, the 
following model is proposed: 


Reaction Time = Part, + Color, + Hue, + Noise, + Image, + error, 


iklmn 
where Part, represents the ith participant of the experiment; 

Color , represents the color of the image (natural or monochrome); 
Hue , represents the hue condition of the image (natural or false); 
Noise, represents the noise condition of the image (noise or no noise); 


Image ,, represents each presented image and 


Error, represents for unaccounted variations within the model. 


Cc. RESULTS 

To remain consistent with statistical testing employed in Experiment 1, the t-test 
with level of significance of @= .05 was used. The reaction times (in milliseconds) and 
error rates (in percent) for each of the participants (Appendix D) were calculated to 
obtain the reaction time and error rate means for each condition of the experiment. As in 
Experiment 1, reaction time and error rate means were compared to determine participant 


speed/accuracy trade-off effects. Additionally, since error rates were not constant for 


qh 


each inverted condition, bias effects were determined to be non-existent. A box plot and 
line plot of each condition 1s shown in Figures 22 and 23 to provide the reader a rapid 


view of the data. 


Box Plot of Mean Reaction Times for all Conditions 
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Figure 22: Box Plot of all image conditions mean reaction times 
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Figure 23: Line Plots of mean reaction times and error rates for Color and Gray 
conditions with "No Noise" and "Noise.'' 





NATURAL GRAY 
DIFF NC NCNOISE CFH CFHNOISE NG NGNOISE GFH GFHNOISE 


MEAN RT 742.65 817.78 841.66 1003.24 803.94 855.38 899.29 917.00 


SEM 44.72 59.15 50.67 74.33 42.34 47.56 36.38 76.77 
MEANErr 2.22 9.78 8.89 13.33 7.56 20.00 11.55 14.22 
SEM 1.06 2.59 lez 2.84 1.70 22> 2.00 2.14 
Table 9: Tabulated data for image condition reaction time and error rate means 





Where NCNOISE represents natural color with noise; 
CFHNOISE represents color false hue with noise; 


NGNOISE represents natural gray with noise and 


i 


GFHNOISE represents gray false hue with noise. 

By inspection of the box plot, a large difference in each condition’s reaction time 
medians across all conditions is seen. This suggests that participant response rates are 
affected by a particular image presentation. The large variance of the data also suggests 
that participant performance is affected. The interquartile range comparison between 
conditions of noise and those conditions without noise shows that higher uncertainty in 
response times exists for noisy conditions. It is interesting to note that the false color and 
false gray noise conditions that were manipulated for hue are excessively large which 
Suggests a Severe disruption of participant performance when viewing these particular 
image conditions. There are fewer outliers displayed than what was seen in the previous 
experiment, presumably because the data is more dispersed overall. A single participant 
was responsible for every outlier except for one. The mean error rates observed in Table 
9 shows a relationship that corresponds to the mean reaction times. The speed/accuracy 
relationship between conditions with noise and conditions without noise shows that 
participant's reaction time and error rates increased with the addition of noise. This 
implies that there was no speed/accuracy trade-off. For example, given that there is 
approximately a 52-millisecond difference in reaction time between conditions of natural 
gray with and without noise, the error rate for the noisier condition is approximately three 
times that of the no noise condition. Therefore, it is concluded that under conditions 
where error rates are similar, the difference in reaction times between conditions of 
natural gray with no noise and natural gray conditions with noise would be much greater. 
The line plots seem to confirm this observation. This effect applies to the hue and color 


conditions with noise when compared to the same conditions without noise, although not 
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at the same magnitude. 

Before statistical analysis was conducted to determining whether noise has an 
effect on reaction times across all image conditions, the data obtained from images 
without noise was used to validate portions of the model used in the previous experiment. 
The previous experiment showed that altering an image's saturation has no effect on 
participant reaction time but altering the hue of an image does effect performance. 
Comparing the difference in reaction time and error rate means for hue will determine if 
using additional images validates the findings in Experiment 1. Under the hypothesis, 
there is a difference in the reaction times of these conditions. Table 10 shows this 
comparison. 


HUE 


CFHNOISE- GFHNOISE- 
CFH-NC GFH-NG NCNOISE NGNOISE 


MEAN RT 99.01 95.50 185.47 


SEM 25.24 29.68 43.65 
MEAN Err 6.67 4.00 B75 
SEM 1.45 2u82 2.42 

Table 10: Difference in reaction time and error rate means across all image 
conditions with and without Hue. 





It 1s seen that there is a positive difference across all image conditions that are 
manipulated for hue, similar to the results for the previous experiment. This adds 
additional evidence that altering an image's hue causes participant reaction times to 
increase. There is a large difference between the color false hue noise and the natural 
color noise condition. This indicates that not only does changing hue have an effect on 
reaction time, but noise increases that effect. To test the data, a paired t-test using 
Bonferroni's inequalities with an @= 0.05, four samples and a confidence level of 


98.75% was used. When comparing the color false hue versus the natural color condition 
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reaction time (t(14) = 3.9232 p < .0125) and accuracy (t(14) = 4.5833 p < .0125), color 
false hue with noise versus natural color with noise condition reaction time (t(14) = 
4.2491 p < .0125) and accuracy (t(14) = 1.4176 p = .0125), gray false hue versus natural 
gray condition reaction time (t(14) = 3.2129 p < .0125) and accuracy (t(14) = 1.467 p = 
0125), an effect is found. These results validate the findings from Experiment 1. The 
exception is that in Experiment 1, no effect was found between gray false hue and natural 
gray. For an unknown reason, there is a larger reaction time and error rate difference in 
this experiment than compared to the previous experiment. No explanation can be made 
concerning this result. The gray false hue with noise versus natural gray with noise 
condition reaction time (t(14) = 1.3281 p 2 .0125) and accuracy (t(14) = -2.1619 p = 
.0125) shows no effect. 

To validate the effect of color, we compare the differences between the means of 
each color condition with the same images in the achromatic condition as seen in 
Experiment 1. For a condition of color to be a factor, there will be a difference in the 
means across each condition. Table 11 displays the results. 


GRAY-COLOR 
NGNOISE- GFHNOISE- 
NG-NC NCNOISE GFH-CFH CFHNOISE 


MEAN RT GlEZ2 


SEM 16.05 
MEAN Err 53933 
SEM E62 
Table 11: Difference in reaction time and error rate means across all image 
conditions between Color and Monochrome 





It is shown that there exists a difference across all conditions. A significant 
finding in this experiment that differs from the previous experiment findings is that the 


reaction time difference between the gray false hue and color false hue is positive 
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indicating that participants viewed the achromatic format slower. The only explanation 
for this contradiction may be that participants reacted with more caution in selecting the 
proper orientation. An error rate comparison between the gray false hue and color false 
hue shows that the error rate is less ” ae experiment. This delay in image selection 
contributes to the increase in reaction time and reduced error rate. The remaining 
conditions contain a positive value indicating that images viewed with color will enable a 
participant to respond faster. In tests of the data, a paired t-test using Bonferroni's 
inequalities with an a@= 0.05, four samples and a confidence level of 98.75% was used. 
When comparing the natural gray versus the natural color condition reaction time (t(14) = 
3.8183 p < .0125) and accuracy (t(14) = 3.2922 p < .0125), and natural gray with noise 
versus natural color with noise condition reaction time (t(14) = 0.9818 p = .0125) and 
accuracy (t(14) = 3.2859 p < .0125) an effect is shown. The natural gray with noise 
versus natural color with noise condition effect is contrary to the findings in the previous 
experiment. An explanation for this result may be due to the object size restriction when 
selecting experimental stimuli. Without the benefit of large objects, color played a key 
role in segmenting global features contained in the degraded images. The remaining 
conditions gray false hue versus color false hue condition reaction time (t(14) = 1.7642 p 
2 .0125) and accuracy (t(14) = 1.3813 p = .0125), and gray false hue with noise versus 
color false hue with noise condition reaction time (14) = -1.5682 p = .0125) and 
accuracy (t(14) = 0.3974 p = .0125) showed no effect. 

Finally, to compare an image that is spatially degraded by noise to an image that 
does not suffer from spatial degradation, we will determine if an effect of Gaussian noise 


exists. It is hypothesized that there is a difference in participant reaction time and error 


ia 


rates between these types of images. By taking the difference of the means, there will be 
a difference in reaction times between each condition for noise. Table 12 displays the 
results. 


NOISE 
NCNOISE- CFHNOISE- NGNOISE- GFHNOISE- 
DIFF NC CFH NG GFH 


MEAN RT 75.13 161.58 


SEM 23.41 42.27 
MEAN Err 7.56 4.44 
SEM 2.24 rege )| 





Table 12: Difference in reaction time and error rate means across all image 
conditions with Noise and images without Noise 


There exists an effect for noise under all image formats. In fact, there is a huge 
variability in the effect of noise on reaction times and error rate means with ranges from 
17.70 to 161.58 milliseconds and 2.67 to 12.44 percent respectively. The standard errors 
for the reaction time means are very high and inconsistent across each condition. This 
shows that participants view images without noise much faster and more accurately than 
images that contain noise. All conditions contain a positive value that indicates that 
images that do not contain noise will decrease participant reaction time. To test the data, 
a paired t-test using Bonferroni's inequalities with an a= 0.05, four samples and a 
confidence level of 98.75% was used to test the data. When comparing the natural color 
with noise versus the natural color condition reaction time (t(14) = 3.2093 p < .0125) and 
accuracy (t(14) = 3.3712 p < .0125), color false hue with noise versus the color false hue 
condition reaction time (t(14) = 3.8228 p < .0125) and accuracy (t(14) = 1.5811 p 2 
.0125), and natural gray noise versus the natural gray condition reaction time (t(14) = 
1.3831 p = .0125) and accuracy (t(14) = 5.3324 p < .0125), an effect exists. However, 


when comparing the gray false hue noise versus the gray false hue condition reaction 
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time (t(14) = 0.3127 p 2 .0125) and accuracy (t(14) = 0.8993 p 2 .0125) no effect is 
shown. This implies that color images that lack visual clarity are effected by this 
deficiency more than achromatic images. 

To further validate the visual — for this experiment, the second method of 
Statistical analysis is employed in the previous experiment was repeated. A combined 
one-sided t-test was conducted for individual image conditions using an a= 0.5. By 
grouping individual conditions of a particular group (Figure 24) and comparing these 
conditions reaction time and error rate means to those conditions that do not possess the 


groups stated manipulations, the following results are shown in Table 13. 


COMPARING HUES 


Color False Hue Natural Color 
Color False Hue Natural Color and 
and Noise Noise 


EE 
<< ae Sa 

Gray False Hue Natural Gray 
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Gray False Hue Natural Gray and 
and Noise Noise 
Figure 24: Stacked comparison of all viewing conditions 
with and without Hue manipulated 


Difference between Hue and 
non-Hue conditions 





Table 13: Reaction time and error 
rate means’ for images 
manipulated for Hue 


Based on the data provided in Table 13, there is evidence that a reaction time 


(t(59) = 5.767 p < .05) and accuracy (t(59) = 1.5977 p = .05) effect exists. This effect is 


ae 


also present in Experiment | and is validated in this experiment. These results provide 
evidence that manipulating an image's hue will disrupt participant visual perception. 

To determine if a color effect exists, reaction time and error rate means for color 
are compared to the achromatic images. This comparison will determine whether color is 
more beneficial in object recognition than in achromatic images. Figure 25 shows these 
conditions. 


COMPARING COLOR 


Natural Color ‘ i Natural Gray 
Natural Color Natural Gray with 
with Noise Noise 
Color False Hue Gray False Hue 
i a 


Color False Hue Gray False Hue 
with Noise 


with Noise 
Figure 25: Stacked comparison of all viewing conditions 
with and without Color 


Difference between Color and 
achromatic conditions 





Table 14: Reaction time and error 
rate means for images with and 
without Color 


As observed in Table 14, there exists significant evidence of an accuracy effect 
for the reaction time (t(59) = -0.8706 p2>.05) and accuracy (t(59) = 3.9562 ps<.05). In 
fact, as seen in the previous experiment, participants respond to achromatic images 
slightly faster than colored images. This result can be misleading given that it has been 


shown that altering an images hue affects participant reaction time. As described in the 
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previous experiment, by increasing the degrees of freedom by eeaiinine all color 
conditions and comparing reaction time and error rate means against all achromatic 
condition reaction time and error rate means, the effect of hue is reduced by the addition | 
of the other conditions. 

Lastly, to examine the effect of noise, reaction time and error rate means are 


compared. Figure 26 shows these conditions. 
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Figure 26: Stacked comparison of all viewing conditions 
with and without Noise 
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Table 15: Reaction time and error 
rate means for images with and 
without Noise 


As observed in Table 15, there exists evidence of a significant effect for reaction 
time (t(59) = 3.5702 p<.05) and accuracy (t(59) = 4.9897 p<.05). It is also seen that 
images with noise are responded to slower than clear images. 


LY. DISCUSSION 


The purpose of this experiment is to clarify the role of color in object recognition 
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when viewing spatially degraded imagery. It is hypothesized that natural color would 
enhance object recognition for images that are degraded with noise. 

The principle research that questioned the role of color in object recognition was 
conducted by Wurm, et al., (1993). This research used degraded food images to 
determine color's role. It was found that color aided the recognition of objects in blurred 
and unblurred conditions. 

This experiment used natural scenes for stimuli taken from a professionally 
photographed commercial CD. The reason for using natural scenes was to closely 
approximate imagery that may be found on military night vision displays. These displays 
are small which hinders image resolution. The images were presented in either natural 
color or achromatic condition or manipulated by altering the hue. Gaussian noise was 
also added to half of the images. 

Analysis of the data provided a means to validate the findings of Experiment 1. 
When comparing the combined conditions of natural hue with unnatural hue, an effect 
was found. This effect validates the findings in the previous experiment. These 
overwhelming results strongly suggest that the HVS is disrupted when an images natural 
color is altered. Individually, all image conditions of false hue were compared to the 
natural hue conditions. The findings of this experiment validate the findings obtained in 
Experiment 1. 

When the combined conditions of color were compared to the achromatic 
conditions, an effect was found. This effect validates previous experiment findings. 
When comparing images that are achromatic with those that display color, either false or 


natural, color appears to play an important role in object recognition. Individually, all 
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image conditions of false hue were compared to natural hue conditions. The findings of 
this experiment validate the findings of Experiment 1. When a natural color scene is 
modified by altering the color, independent of image clarity, the HVS is negatively 
disrupted by the color change. Hoge the findings in this experiment also revealed an 
effect between gray false hue versus natural gray that contradicts the findings in 
Experiment 1. No adequate explanation can be determined to explain this effect. 

All conditions were individually compared for color and the results differed from 
the color results found in Experiment 1. An effect was found between natural color and 
natural gray conditions, independent of image clarity. The most likely explanation for 
the significance of color in object recognition found in this experiment is that objects 
contained in the natural scenes are smaller and less pronounced as compared to objects in 
Experiment 1 stimuli. A criterion for this experiment was that objects could not 
encompass more than approximately 25 percent of the visual field of view. This 
reduction in object size required the participant to globally process the entire scene 
instead of identifying a particular object. Therefore, spatial segmentation of individual 
objects was nullified and global scene processing occurred. It was shown that natural 
color was beneficial in scenes that required global processing. However, when the color 
is altered, no effect is evident when compared to the achromatic format. The reaction 
times for color false hue were much less than the reaction times for gray false hue in this 
experiment. This result is completely opposite to the comparison found in Experiment 1. 
Although not significant, it suggests that any type of color may benefit object recognition 
and the HVS is disrupted less for scenes that are processed globally using artificial color. 


A significant effect exists between scenes that are spatially degraded and scenes 
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that contain adequate resolution. This result validates previous eee on the 
importance of color with degraded imagery. Without the benefit of identifying objects by 
profile and surface features, color (natural and false) is significantly important in the 
segmentation of boundaries. 

The results of this experiment validate previous findings in that altering an image 
by modifying the color disrupts the HVS. Images are more efficiently processed in 
achromatic format than in the unnatural state. For scenes that are processed globally, 
color enhances object recognition. 

Whether observing imagery that is either degraded or clear, the role of natural 
color is significant in object recognition. When the color of the scene is modified from 
the natural state, the HVS is disrupted and therefore is less efficient in object recognition. 
It is recommended that scenes be represented achromatically if natural color cannot be 


attained. 
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We CONCLUSIONS, AND RECOMMENDATIONS 


The objective of this thesis was to evaluate natural and artificial color on natural 
scene perception. Following an extensive review of previous research concerning the 
role of color in object recognition, it was determined that color generally played a 
specific role in enhancing the recognition of objects, but this role was only secondary to 
identification by object profile. It was also shown that depending on the experimental 
methodologies used when conducting the research, the role of color was often 
inconclusive. Some of the deficiencies found in previous research experimental 
methodologies were the inadequacy of target availability. Also, "natural" targets were 
poorly categorized as living, man-made or non-living objects, and the targets were never 
presented in the natural environment in which they belonged. For instance, an apple was 
placed on a flat surface against a neutral background and was either applied color or 
remained as a line drawing. The apple was never located in a tree or placed on the 
ground near the tree to provide other visual cues. Tasks were conducted in a controlled 
fashion, normally a naming or verification task. Other experiments altered the edge 
definition of the objects or provided atypical color schemes to determine color's role. 
Yet, not one of these experiment methodologies applied object recognition in a global 
Seciic. 

The foundation of this thesis was to determine whether the derived research 
conclusions of the role of color could be applied to objects in a global scene. By 
selecting images from an enormous sample of images using strict selection criteria, this 
thesis was able to test the various color theories. The hypothesis developed from the 


research was that when comparing naturally colored scenes to false colored scenes, 
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participant reaction times and error rates would be different. Using professionally 
photographed scenes taken from various categories, it was presumed that a naturally 
colored scene would be viewed similarly as the scene in achromatic format. However, 
when the scene was altered by changing its natural state, the HVS would be disrupted 
when tasked to properly identify the orientation of the scene. 

A second hypothesis tested the role of color and artificial color in a global 
environment that was spatially degraded by noise. Using the abundance of scenes that 
were available to conduct the research, it was hypothesized that color would aid in object 
recognition when compared to achromatic representations of the scene but artificially 
colored scenes would hinder the recognition process. 

The results of both experiments were conclusive. When comparing prototypical 
colored scenes to the achromatic version of the scene using defined edge definitions, it 
was found that natural color did not significantly enhance object recognition. Although 
participant reaction times were slightly smaller for color, statistical analysis found this 
effect to be non-significant. This finding confirms the majority of conclusions found in 
prior research, that color does not benefit object recognition. However, when the color 
was altered to an atypical or unnatural representation, participant reaction time increased 
significantly; therefore it was concluded the effects of color on object recognition were 
significant. This important result is perhaps the most significant reason that 
contradictions exist from previous research. Vision scientists often fail to distinguish 
between natural and artificial color. If given the choice of recognizing an object in an 
atypically colored scene, it was preferred to eliminate artificial color from the scene. 


Additionally, changing the level of brightness to the scene had no effect on participant 
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reaction time for color. 

Natural color becomes more important when the visual acuity of the scene 
degrades. The second experiment validated previous claims on colors role in object 
recognition for degraded images. When compared to the same scene with the addition of 
noise, color improved the ability to respond to object recognition. Therefore, it was 
concluded that color enhances object recognition for degraded scenes when compared to 
a monochromatic representation. 

Current military research in NVD technology has provided the technology to 
combine various scenes into a simple representation to reduce individual component 
limitations and provide potential strengths. However, technology has permitted 
researchers to apply color to image representations that originate outside the range of the 
HVS's sensitivity. The ability to provide standardized colors to the global scene 1s the 
subject of much skepticism that questions this feasibility. The conclusions drawn from 
this thesis provides evidence that color is beneficial when viewing spatially degraded 
imagery. However, with scenes that are not spatially degraded, natural color is more 
effective for object recognition when the scene is viewed globally. 

Further research is required to validate the conclusions of this study. First, the 
images chosen for these experiments were commercially photographed scenes. It is 
recommended that identical procedures be employed with images taken from fused 
sensors. The addition of noise in the second experiment was to approximate the lack of 
visual resolution found in small military displays. By utilizing real-world images taken 
from sensors employed in the field, these approximations would be unnecessary. 


Second, a different task should be utilized to properly identify an object within the 
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scene. Although this type of experiment could potentially limit the number of participants 
that may be available, objects can be chosen that pertain to a participant's particular 
specialty. Therefore, the participant's level of expertise could be tightly controlled. 

Third, conduct the viewing on a display that approximates smaller military 
displays instead of a computer color monitor that contains resolution features not found 
on field equipment. This would add additional realism to field-testing. Finally, by 
varying the degree of image saturation and hue manipulation, a determination can be of 
the amount an image can be altered before participant perceptual degradation occurs. 

The concept of providing color to fused displays will one day enable viewing a 
night time environment be seen as a natural daylight scene. However, much research is 


needed to validate the utilities of these technical achievements. 
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APPENDIX A: COMPUTER CODE FOR EXPERIMENT ONE 
(WRITTEN IN ‘C’) 
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#define _cdecl 

tame lLude <Stalo.h> 
Fimelude <stdlib.h> 
#include <conio.h> 
si riclude <stdadet -h> 
tamelude <strangst= 
#include <time.h> 
#include <vrg.h> 
#include <patch.h> 
#include <story.h> 
#include <keys.h> 
#include <file.h> 


#define FIXATION 1 
#define STIMULI 2 


#define 
#define 


NUMBER_OF_IMAGES 240 
NUMBER_OF_PRACTICE_IMAGES 30 


//keys ‘1' and '2' will stop interval 
//keylist must be NULL terminated 
VstoryKey keylist[] = {{'1', VSTOP},{'2', 
{12> VSTOP)]{ NULL} } > 


VSTOPjen GC 7 Velen. 


int Ran(int max); 


void Permute(int* array, 


void main(void) 
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Uae es alive 
EO ai/ es 
"£043 < 
PEAS = 
“rOSs- 
Me OG 
EOC 
wha Oe eae 


, HOTS. 


PEOSS.. 
“£O091- 
oe OO 
"£103. 
LELOD:. 
El Loe 
efi 
Pete eZ) 
em bcican 
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pox 
ex ac, 
pex", 
pcx 
pcx", 
pcx", 
Pex 7, 
pcx", 
pcx", 
pcx", 
pcx", 
pCcx; 
Pox, 
pcx", 
Dex"; 
pcx", 
pcx", 
pcx", 
Dex, 
Pex ; 
Dex’. 
Dex; 


pn £OOS 


DOL 4 
Eo2 0 
iE OZ'6 
BROS 2 
EOS 6 
"£044 
Tze 18) 
2 LOs6 
ytOVG2 
"£068 
"£074 
"£080 
VE Ve G 
P elObs) 7 
rtog9e 
"£104 
mae IL ALG, 
gaa AR 
spas a 2 


EZ oe 
woes lo 


scx 
{Dez 4 
-pex” 
Spex. < 
EpCx. 3; 
5 OL Ob nee 
=pCx: 4 
ocx | 
~oex 3 
Poe x, 
SDex 5 
Spex, | 
2pex 
4 © ep diear 
Deo 
DCH. > 
speCX., 
SOC. 
Fa @ | oop, ae 
sex; 
Dex), 
pcx", 


rigsel 0) 0s 


eo 15 
5m OVA IL 


si 3 0 
TEO39 
"£045 
i) oe 
PLO! 


PEOO9 
Om =o 


en eal 


EO Se 


uF sig OBS)AS, 
Eds 
ated le 
peal By 
WEi23 
pe Paes 
TEL 


-pCx ; 
“Dex” 
sDexX | 
EpEOZ Ie 
Dex: 7 
Spex. 
2 pCxXuy, 
~pex = 
“Dex, 
TOG Ss 
Spex | 
Pe lep, dame 
Dex | 
fy) ol ep Gear 
Foye One bie 
ADC ay 
~pexs 
~poex | 
Dex; 
Dex e 
apex 
POC, 


nex 


pe, 


pex'"': 


Pe © 
Pea 2 
"£148 
"f154 
Salk eG, 
PELGS 
il 2 
waa kegs: 
"£184 
El 9 © 
elec © 
eZ U2 
oZOS 
"£214 
Terre) 
Liz 6 
BES 2 
"£2359 


char Practice [NUMBER_OF_PRACTICE_IMAGES] [10]={"p4.pcx", 
woe OC MD) DCX. , Dpeepex , "pDIspex ,' plLO.pcx”, pil. pex, 
LplS. Dex. pl Oepex  ,"DL?y pex , pic 
[p22 pex., Pace pex-, P25-pCx , pad 
bDi2i Pex Oo AOe Dew. DZoepexs.,5. Do0 
5338 pCre |; 


Moxie: 
"p19 
eas) 
Ne \siak 


char 
mals 
TS le 
ste i 
“els 
Pole 
Gli 
Me 


Dex, 
Pisenael le 


Poex 


Boek; 
Dex 1 | 
Spex, 
npc X se; 
Dex. 
“Dex, 
“Dex :s 
soem, 
ae Loe 


“W 


epex 


NOC x; 
0} e>, Guar 
ROCK} 
Sol ep alee 
Deku 
DC hua 
Epes; 


pex" 5 
SDCx | 
Spex. , 
POC cy 


Path[ 16] (50) — tua? \\final\ Vee 9d: \\iElnal\\ng\\ "| 
Myeinal \\ Eh, de \V\tinal\\ViEs\ vee 

MerINabV rev ys Go \\rinal\\ihow. 
WVitimalNNtsa\vc.,  d:\\tinal\ \thsaq ya 
NV rlinal Nunes aN waneal wing.) , 
Nitinal\\uEh\ Veo \ Veimal wu is\ ve 
WreEinal \Wuths\ee ds \\Viamalyvuwelhg\\" - 

eriha lw SdN weer \banal Vuurnscgy\"} > 


jade 


EAS 
ELS 
Pi low 
bee Gr, 
ely s 
geal es) 
yeast" 
gp ol 


Ve 203 
“EZOO 
Viz 
eae esa 
SE227 
ees s 
SEPA RS, 


Pye e ej ovep's a 
wpe pCk", 
"DZ6N Sexe 
LOS2 . Dems 


Dex”, 
_pCcx= 
-pCxXme 
Spex, 
pDGsxs 5 
-pCxX ae 
pCa 
SDCoe, 
Spex is 
“Dex. ; 
1DeEx. | 
sDexs 
-DCxwar 
[Pex 
sDex-) 
Poop, aaa 
2) 2) a ae 
spex”, 


char ThisImage[100]; 


me Cc 


int Order [NUMBER_OF_IMAGES], 


Vpatch_display_sequence_description list[2]; 


[16] 


Paden 
"£144 
PE Loo 
pr Ue. 
pete 2 
‘ype dk tayte: 
wel74 
pe else 8 
BEES 6 


-DpCcx , 
-pex ; 
PpCcx”, 
aeCcK', 
~Dex, 
Dex. 4 
_pexen £175 
AS) a, 
-Dpexkee 
gees 2. 
ped hc kom 
peo O a. 
pee, 
Pe 26. 
E22 ez 
tE22 3% 
vie ota 
wEZA0. 


pcx, 
pcx", 
pcx”, 
Dex, 
pex", 
Pex; 
pexy 


Dpexe , 


pe eR 


sash al Be 
poaskede 
Pr Sy 
Pros 
BEAGY 


"£RC1 
Sil sy 
pe ale 
PETS 
yaya ORs) 
peers ila 
Ue ey 
eZ 
pecZ? 
ea Pe) 


Dex) 7; 


pex- ; 


AOL o>, ale 
OC iu 
SOCK 
“Dex; 
Sexes 
Dex; 
<_DCxw, 
Spex. 
“DCX; 
eDCGx sy 
-pex | 
Oe. , 
,pex , 
apex”, 
“Dexa, 
-pex”, 


"£140 
"£146 
Leo 
pogilsys, 
“f£Lo4 
sep eB A 8 
passage 
"£ io 
ba Biel lets te 
"£194 
"EZ OO 
"E206 
Ee deZ, 
mi eee 
"£224 
T E230 


rie J Ol 


WOO Ue OU Our 0 0, 07.0707 05 0 O00 0 ).e 


want (VEULL) -; 
Vpatch_init (NULL) ; 


Vpatch_realize_message (VDISABLE) ; 


hor { 


Order [y] 


Permute(Order, 


y=0; 


Thesis Lal - 


NUMBER_OF_IMAGES) ; 


y<NUMBER_OF_IMAGES; y++) 
Y; 


Prantt("\nerying to open file\n"); 


gene ({ 
{ 


fp = 


Vpatch_read(fp); 
Vfile_close(fp); 
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Vfile_open("shw_imgs.stm", VTEXT, VREAD) ) 


POC G 
Bo (op <r 
specs, 
pOCxk <a 
’DGx, 
M olep. cay 
Lex 2 
DCX 2, 
soex 5 
Cos 
AEeCx", 
Dew, 
Deka 
SDC a, 
~pex oe 
nOCooy 
pcx", 


I= 


ia ex 4 
Vea. Cx" s 
Yt LS 3eeex 
TEES Ieper sy 
to S exes 
NE. pes 
Fl 77a pex.- 
"eS 3 Cx "| 
"TI S9R oer 
Pe SOG Kk, 
"£201 spe 
"£207 -pex = 
it oe Dew 5 
hn OP) Nike oy ofa cue 
Dh Sexy 
Dies pe a 
“E25 0 pex = 


Bois pes. - 
P egy ~pexs ; 


~pex 3 
DC 
ne ey 


NULL) 


} 

else 

{ 

Drintt("\nGCouldn’t open the 2=stm file, made.) -; 
Getena; 

} 


printf("\n\nWhat are this subjects initials?\n\t"); 
gets (subject); 


strcat(logfile, subject); 
strceat (logimie, ".dat"); 


fp2=fopen(logfile, "w"); 


listlO)einterval 1d 7] FLXATION: 
list[(0].keylist = NULL; 


lasttl) santerval sd) =-sTIMULI: 
list[{1]).keylist = keylist; 


Vpatch_realize(FIXATION) ; 


//practice 

for (y=0; y<NUMBER_OF_PRACTICE_ IMAGES; y+rtt) { 
printf("\nmPress any key when ready to initiate trial");getch(); 
j=Ran (16) ; 
strcepy(ThisImage, Path[j-1])); 
strceat(ThisImage, Practice[y]) ; 
Vimage_set_file(STIMULI, ThisImage) ; 
printf("\n%ts", ThisImage) ; 
Vambeenval=set_background pct (STIMULE, 50.2,50-2, 50.2); 
Vpatch_realize(STIMULI) ; 
Vbeep () ; 
Vpatch_display_sequence_wait(list, 2); 
latency = list[1].nframes; 
response = list[1].key; 
pee Gass) 
1f((response-48) ==2) 
Verror_beep(); 
LE eas) 
if ((response-48) ==1) 
Verror_beep(); 
Vpatch_unrealize (STIMULI) ; 
if (response=='q') { 
DEINE: ( (ner ang “coma t."): 
break; 

} 
printf£("\npractice trial number %d", y+l1); 


onl 


for(y=0; y<NUMBER_OF_IMAGES;y++) { 
printf("\nPress any key when ready to initiate trial") ;getch(); 
Thasctraral = Orderly: 
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dot j=Ran(sepyeewbile (efj—1] >= 15); 
strcepy(ThisImage, Path[j-1]); 
strceat(ThisImage, Image[ThisTrial]); 
Vimage_set_file(STIMULI, ThisImage) ; 
e[j-1]++; 

printf("\n%s", ThisImage) ; 

Pprinttie( ro. oanta’ ;- yr); = 
EOrInct | bp2, vers”, Thisimage) ; 


Vinterval setebackqrouna pecelSlTIMULI, 50.2,50.2, 


Vpatch_realize(STIMULI) ; 


Vbeep (); 
Vpatch_display_sequence_wait(list, 2); 
latency = list[1i] .nframes; 

response = list[1].key; 

ee (59) 


if ((response-48) ==2) 
Verror_beep (); 

if (j3>8) 

if ((response-48) ==1) 
Verror_beep() ; 

Vpatch_unrealize (STIMULI) ; 

if (response=='q') { 
PeincmeC \itr ying to quit") >; 
break; 

printf("\ntrial number %d", y+1); 
Porancr(fp2, “\tela";, Latency) ; 
we (<9) 

Beritice(fp2, “\til); 

ae (7>8) 

Bpmimeer (fp2;, “\E2") > 

Eepiiiatt(fp2, -"\tte", response) ; 

} | 
printf("\nThe experiment is over"); 
Vpatch_unrealize (FIXATION) ; 
Vpatch_unrealize(STIMULI) ; 
Vpatch_destroy(); 

fclose(fp2); 

Vexit(); 

} 

int Ran(int max) 

//generate a random number between 1 & max 
{ 

double x = RAND_MAX + 1.0; 

Pace Vy: 


y= | + rand() * (max/x); 
meturn (yy); 


} 


void Permute(int* array, int length) 

//will permute a an array of length cells 

{ 

int a, z, temp; 

for (a=0; a<length; at+) 

{ Zz = Ran(length - a) + a; 

Z--; //subtract one so y can index an array 
temp = array[zZ]; 


28 


Secor 


array | Zz] 
array (a] 
} 
} 


array{a]; 
temp; 
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APPENDIX B: COMPUTER CODE FOR EXPERIMENT TWO 
(WRITTEN IN ‘C’) 


o> 


#define _ 


#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 


cdecl 
<stdio.h> 
<stdlib.h> 
<conio.h> 
<stddef.h> 
<String. i> 
<time.h> 
<VEg .la> 
<patch.h> 
=<story n> 
<keys .h> 
<file.h> 


#define FIXATION 1 


#define 
#define 
#define 


//keys 


{72-9 


AS) 


i 


STIMULL 2 
NUMBER_OF_IMAGES 240 
NUMBER OFS PRACTICE IMAGES 30 


and '2' will stop interval 
//keylist must be NULL terminated 
VstoryKey keylist[] 
VSTOP}, 

{gq VSTOP}, 
VSTOP}, 
{NULL } } ; 


int Ran(int max); 
void Permute(int* array, 
void main(void) 


{ 

ShObEl ex 2, (Counter 0 - 

char response, answer[3], 

long latency; 

FILE “fp; 

See DE) Shain ag @ Pe 

Gay 518) o 8} 5 

char Image [NUMBER_OF_IMAGES] [10]={"£01.pcex","£02.pcex", "£03 .pex", 
ROD era at Ose Dex ins LOO. DCX," t07.Dex.. 1 LOS 
LEU pes sil pex 4 tl? pcx, £13 .pex”, "fia 
DE One Cx a hime pex ., fe. pex: 7 bio pex "E20 
wi 2Z per jae pex -'tZ2a pee. t25.pex' 2 £26 
La ome we jy eo eK 9 ESO .DCx' ) Lol apex 4 (Loe 
CEoaM mex) yer 39..Dex- , 6350. DExX 4 65) uppers Loo 
PEAUe ex |, -ba)  pex">  t42. penx". 143 .peow. Lad 
ESO pex’, i447 .pex' > “£48. pex" ) bag pex 7 Loo 
(LO. Pex Loo Dex) Lode 77. bo ome 7 ako oO 
"NEDO up yes LOCK.) OU DCX , Lome peek, hoe 
VELOC. LOo. Dex £OO. DCsHt, 1 Lor Dex. LoS 
Die Om Cae an £7 lh pew 7) 2. Deny fio weed. lene 
“EDs Dex . "E71 2pCxX sb loo pex, «fue OC aro 
"G2 .DCKoy Feo - DCX 7 Pele DOx | 6 om Dex ao o 
“EOo pCxX =; sEOo Dex 7 EOUsD Ci | mone Dee. “Ueto 
VEO4 2peK" 7 GE9S “Dex 7) hI 6 POX” Eo. Oe saa wes iS 
"LOO jpex;. "EhOde per) ot LOZ Dex weemG er pese, 
“fr OS2pCx" 7" ELOG2pCx" 7 flO)  pex Pag nO cmpesc 
Ure Capex" y-" fli pex Brit 2 pCks le opera 
WEIS. pex's “Plt6 -pex". “Lili 2pex. eet le pee 
"£120-pcx” ; VEL2U Spex s ER22 Pex treme er 


ak 


ae 


VSLOP js 


int length) ; 


SuUbTece (lO, fogrile( 20) ={ "jv4_0 }; 


96 


-pex | 
specx =, 
“DCxX., 
“pox. 
~Dew 
PpeCx.s 
“pox; 
MOC my 
DEX | 
~pCxK s 
"peek | 
spCx. | 
i) OX @o> tiger 
“poem, 
Be) ei ar 
eDC xa, 


"£104 
panei GS, 
wE LE 
PEAS 
a lr 


Oo 
ge ats se 
ie Ge 
Dera ee 
Hs Clo 
PES o. 
mia: 
ps eo pl 
up eh sae 
EOS. 
pack oye ke 
as ae Bo ae 
ye eet Ee 
sas to Ta 
Oe a 
She |S a 


SOC eG, 
soe x 2, 
LOCK 4 
pe 4, 
apex s 


pcx", 
Dex, 
pcx", 
Dex > 
pex", 
pcx", 
Dex. , 
Dex, 
Des, 
pcx", 
pcx", 
Dexa, 
Dex, 
pex", 
Dex, 
pex", 


ete 
30 
wi 35 
"£140 
"£145 
Pies. 
Pais 
pomallene, 
2 pmaally Ss 
PE? o 
i ae Oa 
i 0 
ees 
pais He 
Tile Be: 
EZ OO 


519) o>, Sage 
SOCK, 
ocx. 
= Dex | 
Soex a, 
- DGaaty 
spCx 
sDex <4 
AS OL op ae 
“Dex es 
~pex.; 
»pex 
BOGS, 
DCX 
sOCKX; 
7S) Paar 


fie PAS 
Lo qiec al 
136 
rele] 
"£146 
BEITSS 
eELS6 
nba sal 
"EGG 
“sale gil 
reg 


Ae or 


pellrsas) 
Pomilskl 
oS 
Pere Ol 


spCx | 
Deca, 
“pex", 
ex 
OC: | 
BDCxX” , 
MOCK, 
POC, 
hp Cx, 
-pex , 
DOCK; 
Dex t: 
SOCK 
per’, 
apex. 
Spex a; 


ee tes). 
pea 32. 
pe esc aya 
wELaZ. 
pel ye 
po iba Ae 
alge ito 7 8 
HEEL GZ. 
cee a 
EA 2, 
ge 
GS. 
ot Le 7.. 
he Edeo 2. 
ial 
TEZO2. 


pex", 
pcx", 
pcx", 
pex", 
pcx", 
pcx", 


DpCX Sar 


Pex = 
pcx", 
Pex, 
DEX, 
peEx; 
Pox; 
Pex; 
pcx", 
PCx =, 


PE ee 


pea Eee 
pda aides 
"£143 
"£148 
ta BESS 
Pe lsc 
PLUGS 
"£i6e 
es aed 
ym ge: 
agile 
ot os 
se me Ye 
wE196 
vEZo3g 
LEZOS 


Dox, 
|S eb, cig 
~OCe 
spex ie 
Lexa 
By © Op aay 
|e) op, auger 
—pCx a. 
‘we 
-DCtue 
Cx 
SOC: 
aoe; 
-pCcses 
1 Oley, Gage 
pcx’, 


is AUIS 
Peus4. 
aS 2. 
"£144. 
"£149. 
pow lia) 
germal Go) Be 
ie Ga 
‘EGO 
is el 
gas lye 
"£184. 
“Liooe 
"£194" 
alg bs Be 
"£204. 
wee 


TECOUMpexae 
SPE Zee Dex 
ig 7d Ua BY eb ca 
Lia . Cau, 
“E22 1)... pee 
PE232 Coa 
WEZS TU .OCsC 


2pexX 
pera, 
-DpCxX | 
nn) OL Ow, gaa 
LEZ2 Oe Cx 
YE2Z3em pex", 
tee So pex 5 


E205 
Pez LO 
pee 1S 
Sf 220 
Mao Tas. 
E20 
vie 5 
"£240 


7 E200 
pec i 
eo rhig 
ee 22 
LEZZO 


ASI ep. aaa 
sDCx "3 
wDCK | 
Loe eer, 
BS) ©) cp, cae 
-pCx™™ 
ey Oop aa 


sDpex \ 
SOC x us, 
nOCK = 
HpCcx” ; 
spe x., 
(Dex) bao. 
noCx! 7 E Zoo 
sOex 7 


"E214. 
SE2I9: 
WEZ24 
wre Zoe 
EZ oa" 
"E2397 


ee Pe 
Be AN 2) 
Yea 


Practice [NUMBER_OF_PRACTICE_IMAGES] {10]={"p04.pcx", 
‘Dek 7 DUO. PCxe = DO). Dex - Duc. DCX’ , "p09 -pex | spl spe x. 
'pCK Uple.. pEx17- bls. Dex spl anpex” "pil pcKk', plewoex. - 
(PCS pO. Pex Dil DCKX! Ge Pe UspGxX” upc ieepCxK' 7 DA22epex | 
[DCw sume + Dec nmere o- OCK =, D260 Dex | pZ/ -pCX”, "p2e.pecx" | 
SDCK AMP oO De <a l= DPexX.: 7o-DUZ .pGx 5 p03. pcxk" }: 


char 
eC 
jegial 
“pel 7 
he PAS) 
Boo 


eWemeroachieroe) 1 S0]= {“d=\\ianalZ2\\nevy, "a: \Weimal2\\ng\\ 2 
Were inal2\\VEC\VN 7 G2\\ fanalZ\ vig... 

eal. \\tinal2\ \noise-c\\";, "dz: \\ftainal2\\noise.g\\", 

ua \\tanal2\ \nolse fey", “d:\\Eanal2\\noise. tq\\",; 

wae NE inal 2\\une\\ 7s Gas Vinal ung \ \ "4 

MG wevrinal2\\uEC\\" » "ad: \ \Veanal2\\utgny”, 

Bae WV Lanal2\y\unoise c\\", "d: \\Nbiamal2\\Vunerse g\\"; 

Wa: \\final2\\unose_fc\\","“d:\\£inal2\\unose_fg\\"}; 


char ThisImage[100]; 

ewe ie lee! af On Or0) oO Or, O40 ONO, O00 10. 0r 0, 0): 
int Order [NUMBER_OF_IMAGES], ThisTrial; 
Vpatch_display_sequence_description list([2]; 
Vinit (VFULL) ; 

Vpatch_init (NULL); 

Vpatch_realize_message (VDISABLE) ; 


for (y=0; y<NUMBER_OF_IMAGES; y++) 

Orderly] = y; 

Permute (Order, NUMBER_OF_IMAGES) ; 

Pointer ( \nteryindg to open file\n"); 

1f ((fp = V£file_open("shw_imgs.stm", VTEXT, VREAD) ) 
{ 

Vpatch_read(fp); 

Vfile_close(fp); 


!= NULL) 


oF 


} 

else 

{ 

printf("\nCouldn't open the .stm file, dude.");getch(); 
} 

printf("\n\nWhat are this subjects initials?\n\t"); gets(subject); 
strcat(logfile, subject) ; 

strceat(logfile, ".dat"); 

fp2=fopen(logfile, "w"); 

list (Oleanterval  1a0— Gel xAl tol 

list (i) eanterval®id = STiIMUEs- 

list[1].keylist = keylist; 

Vpatch_realize(FIXATION) ; 

printf("\nPress any key to begin practice phase") ;getch(); 
//practice 

for (y=0; y<NUMBER_OF_PRACTICE_IMAGES; y++) { 

J=Ran (lo); 

strcepy (ThisImage, Path[j-1]); 

strcat(ThisImage, Practice[y]); 
Vimage_set_file(STIMULI, ThisImage) ; 

printf("\n%s", ThisImage) ; 
Vinterval_set_background_pct (STIMULI, 50.2,50.2, 50.2); 
Vpatch_realize(STIMULI) ; 

Vbeep (); 

Vpatch_display_sequence_wait(list, 2); 

latency = list[1].nframes; 

response = list[1].key; 

ae (<9) 

if ( (response-48) ==2) 

Verror_beep() ; 

fase Ss) 

if ((response-48) ==1) 

Verror_beep(); 

Vpatch_unrealize (STIMULI) ; 

if (response=='q') { 

jepqnialicie (a gleiay siete mere mero nent) ) 

break; 

} 

DuinmeE("\npractice trial mumber ta" v+L)- 

} 


printf("\nPractice phase over, press any key to begin experiment") ; 
getch(); 

y=0; 

x=0; 

z=0; 

for (y=0; y<NUMBER_OF_IMAGES;y++) { 

counter++; 

1£ (counter%30==0) { 

printf("\nTake a break, press any key when ready to continue"); 
getch(); 

} 

ThisTrial = Orderly}; 

det j=Ran (16) }) while (els bh) >= 15) 

strcepy(ThisImage, Path[j-1]); 

strceat(ThisImage, Image[ThisTrial]); 

Vimage_set_file(STIMULI, ThisImage) ; 
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Sal); 

printf("\n%ts", ThisImage) ; 
Epmintee (Los ya Vlei. VEL): 
EpGinet (fo2,). \tos peo nisimage) ; 


Vimcerval set. background pece(STIMULI, 50.2,50.2Z, 


Vpatch_realize (STIMULI) ; 


Vbeep () ; = 
Vpatch_display_sequence_wait(list, 2); 
latency = list[1].nframes; 

ete (a) 9.) 


if ((response-48) ==2) 

Verror_beep(); 

wee (je 6) 

if ((response-48) ==1) 

Verror_beep(); 

Vpatch_unrealize (STIMULI) ; 

if (response=='q') { 
Bret ( T\nervinglseo quilt" )- 

break; 

} 

Prince ("\ntrial number @d", y+i); 
Poder (fp2, “\tsli*, Latency); 
Fey <9) 

Perimet(fo2,)*\tl")- 

hej =o) 

force (fp2, 1° kt"): 

fprintf(fp2, "\t%c", response) ; 

} 

printf("\nThe experiment is over"); 
Vpatch_unrealize (FIXATION) ; 
Vpatch_unrealize(STIMULI) ; 
Vpatch_destroy(); 

fclose(fp2); 

Vexit(); 

} 

int Ran(int max) 

//generate a random number between 1 & max 
{ 

double x = RAND_MAX + 1.0; 

ont vy; 

w= 1 + rand() * (mazx/x); 

return (y); 

} 

void Permute(int* array, int length) 
//will permute a an array of length cells 
{ 


int a, z, temp; 


for (a=0; a<length; att) 
{z = Ran(length - a) + a; 


z--;//subtract one so y can index an array 
temp = array[z]; 

array(z] = array([a]; 

array[a] = temp; 

} 

} 
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APPENDIX C: EXPERIMENT ONE REACTION TIME AND 
ACCURACY DATA 
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SUBJ 1 
SUBJ 2 
SUBJ 3 
SUBJ 4 
SUBJ 5 
SUBJ 6 
SUBJ 7 
SUBJ 8 
SUBJ 9 
SUBJ 10 
SUBJ 11 
SUBJ 12 
SUBJ 13 
SUBJ 14 
SUBJ 15 
SUBJ 16 
SUBJ 17 
SUBJ 18 
SUBJ 19 
SUBJ 20 


Mean RT 
SEM 


NC 
486.47 
622.97 
Bide. 
450.42 
1066.24 
513.68 
562.00 
497.42 
613.09 
845.22 
463.51 
586.99 
533.68 
504.56 
799.68 
712.54 
552.00 
469.46 
547.56 
616.42 


601.05 
34.05 


Color 


CFH 
601.43 
782.38 
848.55 
543.12 
GG 5 
554.78 
615.31 
536.69 
804.68 
1232.84 
504.56 
604.25 
641.41 
631.41 
873.46 
624.75 
813.96 
488.00 
678.30 
729.20 


740.03 
Oo s 


CFS 
483.74 
582.54 
594.21 
479.57 
888.34 
Sea ey 
9D 77 | 
543.12 
708.61 
1394.44 
459.94 
631.80 
710.43 
538.25 
rey) | 
613.45 
570.61 
543.24 
714.60 
584.38 


664.76 
See 


CFHS 
Soom 
TAD 6! 
660.63 
483.14 
hoz Oa) 
62975 
677.71 
538.89 
693.31 
1286.15 
486.12 
567.63 
Syl 
588.87 
852.04 
648.55 
598.09 
577.04 
635.86 
778.86 


726.20 
62.55 
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NG 
473.62 
59537 
673.06 
480.17 
847.74 
SLO 
600.32 
632.44 
742.56 
oe ontis 
453.15 
SLOSS 
798.01 
Sule 
741.37 
614.75 
657.48 
537.61 
590.24 
1089.84 


656.52 
38.13 


Gray 


GFH 
503.69 
740.81 
636.65 
15.90 
1332.24 
66) 
700.32 
542.56 
TAI 3S 
WW PGA 
407.53 
499.11 
701.00 
503.64 
862.16 
603.93 
580.54 
501.31 
638.85 
679.86 


613.75 
53.86 


GFS 
Sie) a)! 
602.32 
710.83 
488.14 
1625.54 
488.14 
615.03 
489.09 
128.57 
OS Sy 
461.35 
SOS 55 
693.95 
485.92 
864.40 
713.41 
634.19 
515.07 
580.32 
657.48 


680.50 
60.20 


GFHS 
513.49 
663.62 
790.07 
505.16 
1396.24 
559.59 
683.66 
Shee 
623.24 
T783e22 
468.56 
605.06 
653.63 
555.55 
1180.30 
673.45 
635.46 
574.77 
549.19 
ST 


14552 
74.94 





Color 

NC CFH 
0.00 
leo5 
0.00 
0.00 
0.00 
0.00 
0.00 
6.67 
0.00 
6.67 
6.67 
13.33 
6.67 
0.00 
6.67 
0.00 
6.67 
20.00 
6.67 
1333 


55 
ee 3} 


CFS 
0.00 
0.00 


CFHS 
6.67 
6.67 


0.00 13.33 


6.67 
6.67 
eee 
6.67 
0.00 
0.00 
0.00 
6.67 
Ib3s 
6.67 
ILS), 838 
55.58 
6.67 
0.00 
6.67 
6.67 
85.338 


7.00 
1.78 


35 
6.67 
0.00 
6.67 
6.67 
6.67 
0.00 
6.67 
6.67 

20.00 

I-33 
6.67 
6.67 
0.00 

26.67 
0.00 

20.00 


8.67 
Eek 
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Gray 
NG GFH 


GFS 
0.00 
eros 
0.00 
0.00 
6.67 
0.00 
20.00 
6.67 
0.00 
Ss 535) 
6.67 
6.67 
13.33 
0.00 
S218 
6.67 
0.00 
20.00 
0.00 
6.67 


6.67 
1.53 


GFHS 
6.67 
0.00 

eos 
6.67 

13.33 

S28: 
6.67 
6.67 

26.67 
6.67 

20.00 

26.67 
0.00 
0.00 

138 

13:33 
6.67 

53.55 
6.67 

40.00 


13.00 
2.49 
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APPENDIX D: EXPERIMENT TWO REACTION TIME AND 
ACCURACY DATA 
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SUBJ 1 
SUBJ 2 
SUBJ 3 
SUBJ 4 
SUBJ 5 
SUBJ 6 
SUBJ 7 
SUBJ 8 
SUBJ 9 
SUBJ 10 
SUBJ 11 
SUBJ 12 
SUBJ 13 
SUBJ 14 
SUBJ 15 


MEAN RT 
SEM 


NC 


687.19 
AS, 
521.41 
510.54 
720.48 
Gis 
802.19 
584.59 
1028.53 
698.31 
815.44 
596.64 
807.19 
687.19 
879.54 


742.65 
44.72 


NC 
NOISE 
733.30 
1402.17 
723.58 
613.17 
720.8 
754.46 
808.3 
621.09 
1220.51 
660.09 
864.85 
57519 
799.41 
5ey8 
1013.43 


817.78 
seh ils 


CFH 


888.3 
1283.84 
671.4 
52579 
745.11 
13057 
934.58 
652.2 
1145.79 
eS 
125,52 
706.38 
855.32 
888.3 
856.98 


841.66 
50.67 


CFH 
NOISE 
Is yeabe 
N22 ae 
O59 2 
673.58 
769.14 
1250.78 
le 2 eal 
(2205 
1521.96 
848.68 
883.99 
(aS7 
927.94 
1113.42 
GS 55 


1003.24 
74.33 
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NG 


13055 
ZIG? 
613.44 
505.93 
1aN2Z5 
Toe, 
919.4] 
679.83 
eee be. 
PSlesZ 
849.37 
745.48 
888.65 
730.53 
932.34 


803.94 
42.34 


NG 
NOISE 
806.52 
1040.24 
626.36 
791.64 
O97 
S555 
880.27 
621.77 
1300.78 
821.12 
745.8 
754.83 
1046.93 
806.52 
1056.87 


855.38 
47.56 


GFH 


978.81 
1154.95 
683.95 
776.08 
797.88 
MS: (ove 
a3 
769.41 
1147.18 
801.25 
PST Te 
751.08 
847.88 
978.81 
901.23 


SO0rz9 
36.38 


GFH 
NOISE 
783.3 
1574.34 
862.84 
631.04 
669.97 
7142.28 
893.55 
676.9 
1513583 
874.32 
940.24 
608.9 
1186.62 
783.3 
1013.54 


917.00 
10mg 





CFH 
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CFH NG 


20 
3595 
13.33 


J Ss) 
2.00 


GFH 


6.67 

6.67 
26.67 
26.67 
26.67 
eee 
Wee) 





2.14 
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